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This report focuses on three aain guestionsi (1) To 
vhat extent does a, Head i^tart experience accelerate the rate at vJbicb 
disadvantaged preschooleos acguire cognitive skills? (2) Are the 
Planned Variati6n models, siaply by virtue of sponsorship tore 
effective than ordinary nonsp^onsored H<i|id Start pl^ograas? and (3^ Are 
soae Planned Variation aodels particularly effective at iapartlng 
certain skills? : The first chapter gives an overall picture of the 
Head start Planned Variation study, irhile the second chapter 
suiiarizes data concerning background characteristics and 
distribution of test scores. Chapter 3 provides a general discussion 
of aethodological issues and soae of the aajor difficulties resulting 
ftoa the study^ design. Chapters 4*7 atteapt to present h picture of 
the pattern of^ overall effects of various prograas throngji ranking 
analysis* residual analysis^ analysis of covariance, and resistant 
analysis. The final chapters ^xpldre the guestio^n of v^ether t|ye 
relative effectiveness of . various prograas is related to certain 
child background characteristics, such as sex, ethnicity, age, priot 
school e:iiperience , and'aother*s education. One lajor conclusion drawn 
as a reniilt bf^the lnteraodol coaparisonfi vas that Head start , 
prograalci are guite hoaogeneous in their ability to proaote general 
cognitive developaent. (CS) 
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INTRODUCTION 



Chapter I 




Background 



In 1965 Project Head Start was initiated with great 
fanfare and optimism. It was thought that since the j 
"disadvantaged" child. arrives at sehool handicapped by 
an educationally impoverished home environment^ he starts 
out behind the middle class child jLn terms of basic cog- 
nitive and socio-eraotional development. This initial gap 
is then propagated throughout the child's sphool career, 
leading ultimately to large deficits in educational attain- 
ment and career success. It was hoped that a summer-*long 
or year- long preschool compensatory program would give 
disadvantaged children the "head start" they need to start 
off school on, an equal footing with middle class children 
and progress from there on at a comparable rata. 

The basic assumption justifying Head Start, then, is 
that' a limited intervention which alters the child's environ- 
ment at some point can permanently influence his potential 
for future educational achievement. If we accept this pre- 
mise, we must still ask certain key questions* How ext'ensive 
an intervention is necessary to effect permanent change? When 



ERLC 



2. 



In the child's life should the intervention begin, and how 
long rjkust it persist? Around the time when Head Start was 
conceived, there was considerable optimism that basic 
intelligence was malleable even into the early elementary 
years. This belief, coupled with several encouraging 
reports on preschool compensatory programs (e.g., Weikart, 
et al., 1964j Gray and Klaus, 1963; Bereiter, et al., 1965), 
fed the hope that a relatively inexpensive solution to the 
problem of educational disadvantage might be feasible. 

Head Start is a program lasting a few hours per day 
over the course of a few months which attempts to rectify 

\ the cumulative effects of four or five years of deprivation. 

\ , ■ ■ , ■ . .. . • 

\ Early evaluations, most notably the Westinghouse-Ohio 

hational evaluation (1969) showed modest positive results. 

Moireover, there was evidence (e.g., Wolff and Stein, 1965; 

Holmes and Holmes, 1966)' that Head Start effects were 

disappearing in the early elementary years. Head Start 

itself does not appear to be the solution to educational 

disadvantage. It may, however, be valuable as part of a 

more comprehensive approach involvihg a more extensive 

intervention in children's lives. 

As a step in this direction, the Follow Through program 

was started in 1967. 
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By 1969 there were over 170 school didtriota with Follow 
Through programs. Follow Through attempted to enrich the 
curricula of early elementary (g^rades K to 3) programs, 
particularly for children with Head Start experience. By 
consolidating and building upon their preschool experiences, 
the program hoped to be able to influence permanently 
children's chances for success in school. 

According to Smith and Bissell (1970), Head Start 
centers were practically autonomous and programs varied 
greatly. "Although lists of goals and objectives were 
developed by OEO, a laissez-faire attitude predominated." 
When Follow Through was originated, it was felt that care- 
fully designed and Implemented programs based to some extent 
on theories of child learning offered greater hope of 
success. A number of well-defined curricular programs, or 
"models" were developed |)y sponsors , who were Individuals 
or organizations with expertise in early childhood education." 
By fall 19»9> most Follojw Through schools had adopted one of 
these models. Thus, the|re was deliberately, "planned variation" 
in the models implementeVi. By comparing the effects of \, 
these models, informatlo^n could hopefully be obtained on 
what kinds of curricula iproduce what kinds of 



ERLC 



4 



results. Children entering selected Follow Through schools 
during. the years 1969-72 were to be^ tested at entrance and 
followed through grade 3* 

In 1969 the planned variation approach was adopted 
for an experiment in Head Start. The Head Start Planned 
Variation (HSPV) study was to involve 3 cohorts of children 
in programs during the school years 1969-70, 1970-71, and 
1971-72. Several of the models used in Follow Through 
were to be implemented at 2 or more sites throughout the 
country. There were three restrictions on the way this was 
to be done: ^ 

1. Sites were to contain pre-existing Head Start programs 

2. Children in a program must livj^ in an area served 
by a school with a Follow Through program. 

3. The Head Start model must ^e the same ^s the 
Follow Through model in t^at area. 

children in Planned Variation j (PV) programs were to 

• ■ { ' ' ' ■ > . " 

be tested at the beginning of the program in the fall and 
at the end in the spring. Children in some Hea^ Start 
Programs without a Planned Variatip'n model (NFV) were also 
tested, so that a comparison could be made between model 
effects and those of typical non-sponsored Head Start pro- 
grams. In addition, during the final year (1971-72) only, ' 
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a group of "Control** children who were not in any preschool 
program was tested in fall and spring, allowing the possi- 
bility of estimating the absolute effect of a Head Star€ 
program. ^ 

This report is concerned with the dat:\ from the third 
year of the HSPV study, the academic year 1971-72 • As the 
result of improvements in the data collection throughout 
the three years of the study, these data are potentially 
more informative than those collected on the first two r 
cohorts* In particular^ the battery of tests is more 
extensive and hopefully more appropriate for program 
evaluation. Additionally, we have the Control children who 
were hot enrolled in any preschool program. 

We had originally hoped to study the impact of Head " 
Start on both cognitive and socio--emotional development. 
QQ For reasons to be detailed later in this chapter, the non- 
Qt^ cognitive measures used in the study proved unsuitable for 
^[N^ use in program evaluation. This is not a reflection on 
those at the Stanford Research Institute and the Huron 
Institute who designed, the study. Good measures of affec- 
Tfy tive characteristics for preschool children which can be 
routinely administered within the constraints of the Head 
start setting are simply unavailable. According to Walker 
(1972) "until the major theoretical questions and issues 
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are answered within a comprehensive theory of socio-* 
emotional development^ socio-emotional measures for 
young children cannot be meaningfully developed^" This 
rejport will focus ^ then, on the short-term cognitive effects 
of various types of Head Start programs. 

The remainder of this chapter consists of six sections. 
The first outlines the major questions addressed in this 
report* The second contains brief descriptions oi the 
HSPV models* The third describes the study design. The 
fourth discusses the data collected and explains which 
measures we have selected for our evaluation and why. 
The fifth section summarizes briefly the relevant findings 
of other reports in this series. Finally, we present a 
brief summary of this chapter and overview of the remainder 
of the report. We do not^present an overall review of 
the literature on the effects of preschool programs. 
The interested reader is referred to Stearns (1971) and 
White et. al., (1972) . 

Major Questions 

Head Start is much more than a training progra^i to 
prepai^e disadvantaged children to perform better in school. 
It provides a wide range of services, including health care, 
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nutritional benefits, training in social skills, ,arly 
detection of severe problems , and a focus for parent 
involvement with the community. Ideally, in evaluating 
the full impact of a. Head Start program,, we wojald need to 
measure many aspects of an Individual throughout his life. 
Even if we could circumvent the measurement problems and 
practical difficulties in setting up an experiment designed 
to do this, it would take literally a generation to com- 
plete^ by which time the results might no longer be relevant^ 
The questions which we can hope to answer are restriected 
to effects which are relatively short-term and limited to 
outcomes which can be measured relatively well^ i.e* short- 
term cognitive effects. Improving the ability of disadvan- 
taged children to acquire academic skills in school ^ T 
was one goal of Head Start. Although recent work 
by Jencks et al« (1972) suggests that such academic performance 
may not be so strongly related to future financial success 
as many thought, it is generally agreed to be a worthwhile 
and important goal. It can be argued that short-term pre- 
school program effects do not ensure later school success. 
It seems reasonable, however, that a program which sub- 
stantially taises cognitive skills for preschoolers can 
, have a lasting influence if appropriately augmented during 
the /early school years. The study of Follow Through 
currently being conducted by J\bt Associates (1973) should 
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help US Jto Unclerstand to wKat extent this is realistic. 
In this report our analyses will focus on three wain 



luestlons: 



1^ To Vhat jpxtent' does a ^Head Start experience 

V./" ^ ^. ^ ' . \' ■ . . > 

accolefrate tt\je rate at: which disadvantaged pr^^ 

schoolers acquire cognitive skills?' / 

* V f > - • 

2^ Are the Planned Variation models, 'simjjjy by virtu6 . ' 
of sponsorship, more effective than ordinary non- . r 
sponsored Head Start programs?^ ^ ,j / 

3. Are some PV models particulairly effecftiye at * v 

imparting certain skills? , ' . . 

These are JaQsentifllly the same questipns aiJdr^ssed ; . 

by Smith (1973) in his report, pn the 1970~'/1 .$;oWt data 

although the batteryfof tests employed and the statistical 

■ analyses used are different. We will also concerned 

. ■ ••• ' 

with evidence of interactiorfs between program effectiveness 

..and spe<?ific child characteristics. Given the 'limitations 

ifiiposed by the degign, the methodological problem's *involved 

in eliciting such interactions are formidaip'l^. ' Any conclusions 

. • . ' n.'";) ^ ^ 

can only be in the form of suggestions" rather than strojig ' 
assertions. . ^ ' '■. 
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Model Descriptions . 

There were 11 models for which child outcome information 
was collected during 1971-72 ♦ These models may be thought 
of as varying in ternis of a number of dimensions along 
-which preschool programs can be ranged. White et al., 
(1972) summarize > the literature on such classification 
schemes. . 

For our artalysis, one of the most Important of»,. these 
dimensions is the extent to which the acqulsltion'^f aca- . 

demic skills is^stressed through formal, highly-structured 

■ ■ . ' * ' ' ' ■■ ■ — ■^f^^V-"" " 

^activities. Traditional preschools and Head Start centers 

.vary in their stress on such acfelvitiea. Many g:e fleet a 

developmental approach which tries to create a milieu in 

which the child is encouraged to explore and learn from V 

his environment, rather than respond to dejnands leading to- 

cognitive growth in a pre-speclf led way. At least thrge of 

the eleven models are consciously concerned, with the develop-* 

ment of specific academic skills useful in the early school 

years. ^ These are th^ Oregon, Kansas, and Pittsburgh mo^^ls, . 

While the models do vary along certain important dimension 

relative to the. condition of no preschool program their 

similarities, far outweigh their differences. According 
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to Smith (1973) , 

All of them seek to develop children's 
learning abilities. All are convinced 
of the importance of individual and small 
group instruction and frequent interchange 
between children and concerned adul ts . All 
attempt to make learning interesting and 
relevant to the child's cultural background . 
All believe that the child's success .in 
learning is inseparable from his self-esteem, 
motivation^ autonomy, and environmental ' 
support, and all attempt to promote success- 
ful development in these domains while 
fostering academic goals. 



We conclude this section with brief descriptions^ 
taken from Smith (1973), which attempt to give the flavor of 
the various programs. For more complete descriptions, see 
Maccoby and Zellner (1970) and the Rainbow Series, published 
by the Office of Child Development (1972) . 



The Enabler Model 

Office of Child Develofiment 

Sponsor Contact ; Jenny Klein 

The Enabler Model is not really a curricular model. Rather 

it is an approach involving the total community which is built on 

goals prescribed by each community for itself. The development 

and implementation of this model are facilitated by the 

assistance of an OCD consultant who takes a very active 

role in all aspects of the program. Thus projects with 

the Enabler Model may differ considerably in the approach 

and style of their educational tactics, but all share 

a commitment to high levels of staff and parent participation ir 

^)licy making, program planning and classroom operation. 
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EDC Otien Edlication Curriculum 
Educational Development Corporation (EDC) 

Sponsor Contact ; George Hein 

EDC has an open classroom approach derived from the 
British primary school model and theories of child develop- 
ment. It believes that " learning is facilitated by active 
participation in the process. The classroom provides a 
setting in which there is f range of materials and activities 
from which the child can choose. Academic skills are 

* 

developed in a self -directed way through classroom experi- 
ences. The role of the teacher is one of leading the child 
to extend his own work and generally involves working with 
an individual child or small group » 

The Systematic Use of Behavioral Principles Program ' 
(Engclmann-Becker ) 

University of Oregon , - 

S ponsor Contact : Wesley Becker 

The primary focuv of the Engelniann^-Deckcr program 
is on promoting skills and concepts essential to reading, 
arithmetic and language achievement , with particular 
emphasis on remedying language deficiencies. The main 
techniques are programmed; materials , structured rapid- 
fire drills, and positive reinforcements of rewards and 
praise to encourage desired patter'ns of behavior. Small 
study groups of five to ten children are organized by 
teachers according to ability levels in order to facilitate 
presentation of patterned learning materials and to elicit 
^jj-^bal responses from children. 
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The Bank Street ColIe<^e of Education Approaoh 
Bank Street college of Education^ 

Sponsor Contact : Elizabeth Gilkesoh 

The Bank Street approach emphasizes both learning and 
social-emotional development of 'child^ren on the premise . 
that they are intertwined. The teacher functions /as a" 
supportive adult whom the child can trust, and teaches by 
relating and expanding upon each child's response to his 
experiences. The classroom is viewed as a stable environ- 
ment and workroom for the child in which he is' encouraged 
to explore, make choices and carry out plans. Academic 
skills are presented in the context of classroom experiences. 

The Behavior Analysis Approac h 

Support and Development CenteV for Follow-Through, University 
of Kansas 

Sponsor Contact ; Don BUshell 

The Behavior Analysis approach has three predominant 
aspects. First it emphasizes academic and social skills • 
Individualized programmed materials are the primary 
teaching mode. Second it makes systematic use of positive 
reinforcement. A token exchange system is used to support 
children's learning efforts. Third it employs parents as 
members of the instructional team as well as behavior 
modifiers. They receive training and v^ork in the classroom 
in shifts throughout the year. 
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Indtvidualiy Prescribed Instruction and the P r imary Educatio n 
Project (IPT T ~~ 

Learning Research and Development Center, Univ. of Pittsburgh 
Sponsor Contact ; Lauren Resnick * 
The IPI approach provides an individualized program 
of instruction for each child which teaches him academic 
skills and concepts in the areas of language, perceptual 
motor mastery, classification^ and reasoning. The htaterials 
are sequenced to reflect the natural order in which children 
acquire key skills and concepts. Diagnostic tests determine 
each child^s strengths and weaknesses and are used by the 
teacher to prescribe instructional materials appropriate 
to his needs. Positive reinforcement, both social and 
.concrete, is given continually for success in learning* 

T he Responsive Environments Cor p oration Model* (RK C) 
Responsive Environments Corporation 

Sponsor Contact : Lori Caudle 

The REC model uses specially designed/ self-correcting 

multi-sensory learning materials which strengthen school 

readiness skills in language and reading. They are designed 

to teach basic concepts while allowing children to make 

choices, work independently, and set goals for themselves. 

Teaching machines in the form of "talking typewriters" 

and "talking pages*' involve children in learning by seeing, 

tracing, typing, imitating and discriminating among sigh^s 

and sounds and»,by recording and listening to thoir own 

voices . 



The Flbrida Parent Eaucator Model 
University of Plorida ' 

Sponsor Contact : Ira Gordon ' 
The Florida approach is not a specific classroom 
instructional model but is designed to work directly in the 
home. It focuses on thcs parent, believing that the parent 
is the key a^ent in a child's development. The major goals 
of the program are to develop educational cprapetonce in 
the child and to develop an atmosphere in the home which 
Jill foster continued growth. An important role is played 
by paraprofessionals called par^jnt educators. The parent 
educator spends half^timo with the teacher in the class- 
room and the other half making home visits. The home visi't 
involves bringing tasks into the home and . instructing the 
mother how to teach them to the child. 

The Tucson Early Education Model ' 
University of Arxzona 

Sponsor Contact : Ron Henderson 

The Tucson model has a flexible child-oriented 

curriculum which focuses simultaneously on four areas of 

developm^ent: language comp , 

motivational skills and societal skills. Emphasis is 

placed more on learning to learn skills than on specific . 

content. The content is inci\/idually determined by a child 

environment and interests.^ The classroom is arranged in 

interest centers for small groups. The teacher's role is 

to work on a one-to-one basis with the child, arrange the 

^^^ssroom sotting and encourage interactions between the 

™^?fld, his environmo^nt and others. 
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Reapqhslve Edugational Proci;.'am 

Far West Laboratory for Educational Research and Development 
Sponsor Contact s Glen Nitnnicht 

The Responsive Educational model emphasizes self- 
rewarding learning activities and a structured environment 
responsive to a child's needs and interests. The model 
encourages the child to make interrelated discoveries 
about his social woi^ld and physical environment and j 
stresses the importance of the development of a healthy 
self-concept, Vhe classroom is a controlled environment 
in which the child is free to explore various learning 
centers, games and activities; Problem solving and concept 
formation as well as sensory and perceptual acuit^V 
Stressed and the pace of all leaifning activities' is cet 
by the child for himself, 

Qognitively Oriented Curriculum 

Hi/Scope Educational foundation y 
Sponsor Contact : David Weikart 

The Cognitively Oriented Curriculum ^combines Piagetian 
theory and an open classroom approach. It uses a cognitively 
oriented curriculum and emphasizes the process of learning 
rather than particular subject matter. It stresses a 
child's active involvement in learning activities. The 
teacher takes an active role. Additionally, home training 
is seen as part of the program and the teacher suggests 
tasks for the mother to present to the child at home. 
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Design of the Study 

During the 1971-72 academic year ther0 were 29 tested 
Head Start locations* There were 28 locations eo?itaining 
one of the 11 models. Of these PV: locationsi 11 also con- 
tained non-sponsored (NPV) clas8roox()8. One place (Des Moines) 
contained only NPV olassroomst In addition there were three 
places containing groups of Control children not enrolled 
in any preschool program throughout the year* Theue were 
y Huntsville, San Jose, and Sacramento. These children 
were contacted by direct recruitment of from Head .Start 
waiting lists. * \ 

The number|,ng system ueeq to identify locations is 
somewhat complicated • Each location has a four digit code. 
Since each Head Start location is located in an area served 
by one of the Follow Through models, the code used to 
identify it is the same as that assigned by SRI to the Follow 
Through site, with the exception of the Enablers model, 
which is unique to HSPV. The first two digits identify 
t^^^ and the second two identify the site uniquely. 

Thus I 0711 refers to Follow Through site nutt4>^er llih the 
Oregon model (07). The three control locations were 
given codes of 2801 (Huntsville) , 2802 (Sacramento), and 
2803 (San Jose). 

During 1970-71/ each model was implemented In at 
least one location with both PV and NPV classrooms. In 

ERIC 



1971''72 only seven of 



prograiHB for purposes 
models and the Contro 
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the models hais suoh comparisons. 
In most of our analyses we pool all MPV classrooms and 
treat them as a representative sejtiple of non-sponsored 

of comparison with the various PV \. 
children « 

Let us define a i^site" as a group of children in a 
particular location undergoing a particular kind of pre- 
school experience* Thlis, we have 28 PV sites, 12 NPV 
sites, and 3 Control sites, for a total of 43 sites. 
These 43 constitute a convenient set of units of analysis ' 
for some purposes* If this study did nojb e^^st in the con- 

text of other Head Start and Follow Through studi^g, we | 

. " ' ■ ■ ■ ■ ■ 1 • ^ 

might number these 43 sites in some convenient and logical 

^ ' - ■ 1 

way* As it is, we haye decided to retain the old SRI 

coding system. Thus, we are stuck wJLth the awkwardness of 

having, for example, a site 0711 PV and an 0711 NPV* A 

complete description of the design is provided by Tables 

I-l and 1-2* 

For convenience throughout this report we shall often 
refer formally to the NPV children and the Control children 
as inodel or program groups. They are, of course, not 
models or programs In the same sense as the PV models, but 
it is awkward in terms of reporting results to continually 
make this distinction* Thus, from an exj^erimental viewpoint 
we can thinl^pf our 43 6ites distributed across 13 programs 
to be compared (11 PV models, NPV, Controls). 
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Table l-l 



PIANNED VAl^ATION SITES 



Model 






f lestea ► 


Year 


Site 


Code 


Classes 


Joined 




Duiutn 


< 

0204 . 


\ 6 


,70 




Salt Lake 


0209 


6 


69 




Tacomci 


0211 


6 


70 


U* of Arizona 


Lafayette 


0308 


6 


69 




LaKewooa 


0309 


6 


69 




Lincoln 


0316 


6 


70 




Tuskegee 


0510 


6 


. - 
69 




Wilwingon 


0511 


6 


69 




Elinira 


0512 


6 • 


70 , 


u« or Oregon 


Tupelo 


0711 


4 


69 




£• Las Vegas 


0714 


5 


70 


U. of Kansas 


Portageville 


0804 


4 


69 




Mounds 


0808 , 


5 


70 


High/Scope 


Ft* Walton Bch* 


0902 


5 


69 


Central Ozarks 


0904 


6 


69 




Greeley 


0906 


4 


70 


U« OX if xoritia 


jonesporo 


1002 


A 

3 


69 




Chattanooga 


1007 


6 


70 




Houston 


1010 


5 


70 




- ■ 

Paterson 


li06 


7 


70 




' jonnston co» 


1108 


6 


69 


0. of Pittsburgh 


Lock Haven 


1203 


6 


70 


Montevideo 


1204 


4 


7J 


REC 


Kansas City 


2p01 


6 


70' 


Enablers 


Newburgh 


2702 


6 ' 


70 




Bellows Falls 


270 3 


7 


70 
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Billings V 


2704 


6 


70 




Colorado Spring 


2705 


6 


70 



Table 1-2 



NON-PLANNED VARIATION AND CONTROL SITES 



Sito 



Code 



# Tested 
Classes 



Year , 
Joined . 



NPV HS 



Des Moines 
Tupelo . 
W. Las Vegas 
Portageville 
Mounds 
Greeley 
Jonesboro 
Chattanooga 
Houston 
Paterson 
Johnston Co. 
Kansas City 



Control 



Hunts vill^ 

Sacjeaifien^to 
San Jose 



0305 
0711 
0714 
0804 
0808 
0906 
1002 
1()07 
1010 
1106 
1108 
2001 



2801 
280 2t 
2803 



8 
4 
4 
11 
2 
4 
3 
4 
4 
2 
4 
5 



71 
69 
70 
69 
70 
70 
69 
70 
70 
70 
69 
70 



71 
71 
71 
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Data Collec ted # • 

. ■ ■ ■ . s ■ ' 

In this section we describe briefly, all the instru- 
tnents used in this study. A concise picture of the entire 
data collection effort is presented in Table 1-3. Some 

information was collected on the full sample (F) of 

f ' ' ' ' ■ ' • ' ' 

children in a tested classroom. Other information was 
collected on a partial sample (P) . This partial sample 
consisted of a random sample of one third of the children 
in the class unless there were fewer than 18 children, 
in which case 6 were tested. Some tests were given in 
both the fall and springr while others were given in the 
spring only. The test battery for Control children was 
somewhat different from that for Head Start children. All 
^at^ collection was carried out by the Stanford Research 
Institute (SRI) with advice from the Huron Institute • A 
more complete description of data collection activities 
can be found in the SRI final report (1972), ^ 

In the analyses in^ this report, we focus exclusively 
on eight tests as outcome measures, Theffe the 32- 

item Preschool Inventory (PSI) , the Peabody pBSfture 
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Table 1-3 



Data Collection Activities for the 1971-72 Year 
, of the Head start Planned Variation Study 



FALL '71 



, F="Datd for entire class. 
P=»Data for randomly se- 
lected 1/3 of class. 

SPRING "72 



Inotrument 



Presohool Inventoirv 


P 


F 


P 


F 


Vocabulary 


F 


P 

- • 


F . 


F 


Widd RaEnge 3 
^ .AchleveihQ)rit 


P . 


P 


F 


F ■ 


^ITPA Verbal Expres8i6n 


P 




P 




ETS EnuinQration 


P 




P 




8*-Block Sort Task 


p 0 


. : p 


pi—— 


• F 


IDS Seif^Cbhcept 


p 


p 




F 


Classroom Behavior 
Inventory 


p 




F 




~ — T A ^ 

Motor InhibUion " 


p 




P 




Gujnpgookies 






F 




{Relevant Redundant Cue 










Classroom Information 
Poriji 


p 




F 




Parent Information 
Form 


p 


f2 


P 


p2 


Teaelier Information * 
Form 


F 




F 


Mi. 
it: 



^Data collected only at sites: 0204, 0316, 0512, 0711, 0804, 0902, 
1002, 1010, 1106, 1203 

''Diffferertt foxTO for controls 

spelling, oral arithmetic, written arithmetic, word reading 
ytjlL asts given in spring only. 



Vocabulary. Test (IJPV) , four subtests of the Wide Ranqe 
Achieveh\ent Test (^RAT) , the Verbal Expression , sUbtest of 
the Illinois Test of Psycholinguistic Ability (ifPA) , afid the 
^Educationa!^ Testing Service Enumeration Test (ETS) . 

We focus on these tests partly as a result of 

' • ^ * - ^ * ' . 

problems in the way the data were collected, but wore impor- 
tantly because we felt^ there were crippling limitations on 
the usefulness and appropriateness of the other measures 

as eva^uati^^e instruments. In the bri^f descriptions 

*■ • ' . ' < ■, 

presented be low ^ we giv6 specific reasons for excluding 

\ - ^^ . . 
each measure which * is not used in the a'r^alyses. Not 

'-■<-■ ' , ■ ■ r 

surprisingly^ the tests which are suitable all irieasure 

skills in the cognitive domain* We wou^ like to be able 

y . / . ' * ' 

to study effects in tjie ^a^ffecti^e domain, but we have, 
concluded with reluctance that the instruriients used in this 
study nee^d^ further refinement befpre they can b^ relied on. 
Since these instruments are experimental, we felt that it 
would be more valiiable to look' at .the data from the view- 
point of what we can learn a^out the' tests and how they . 
might be useful in future evaluations, rather thah what we 
can learn about program ,e,ffects. These analyses are pr6- 
jsented in detail by Walker et al* (1973) • The test battery used 
during 1971*72 was completely different from that used In. 
1970-71. Thus we cannot replicate previous finding's nor 



.0 ■ 
make comparisons across cohorts. It is hopeO., however, 

that the test battery represents an improvement arid will 

be -a more sensitive detector of program differ;ences, in ^ 

p^^rticular, several of the tests measure specific academic 

" • * / ' ■ • 

skills as opposed to general intellectual ability or ^ 

achievement. Programs rriay^^iffer more in their effects on 

0' * 

s|uch skills. 

" ' .' ' • ^ . ' 

^ ' An important consideration in selecting tests wap the 

fact that a study was planned to follow up many' oft* the 

children in, our study into their first school year in a 

Follow Through program.* Thus tests suitable for slightly 

older as well as pre-school bhildren were 'gouqht, so that 

€he developmental process over at least a two-year period 

could be studied. » . 

We begin our test descriptions vith the eight outcome 

measures selected for use.- All correlations mentioned are 

taken from a table in Walker's report (1973) which weiyhave 

for convenience reproduced here as Table I~'4. These 

correlations an^ based on the fall test results for the 

entire Head ^ Start sample. All other - information quoted 

can be found jLn Walker's report, and we provide no further 

documentation. 



*Thi3 study will be carried out by Abt Associates, beginning 
September 1973. 
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Caldwell Preschool Inventory (PSI) 

The PSI is designed to assess general achievement in 
skills useful for later school success. In 1971-72 a 
32-item version was used, consisting of a subset of the 
items in the 64-itein version .used in 197-71 • Our best 
estimate for a reliability coefficient is ,B3. Correla-- 
tions with all other cognitive tests in the battery are quite 
high, the highest being .665 with the PPV. For the Head 
Start population there appear to be no ceiling (test too 
easy), or floor (test too hard) effects, and the distribution 
of scores is quite symmetrical. With its generally ex- 
cellent psychometric properties and the fact that it taps 
very general information processing skills and preschool 
achievement^ the PSI is potentially the most useful test 
in our battery for program evaluation. Its very generality 
may, however r make it insensitive to program differences, . 

Peabody Picture Vocabulary Test (PPV) 

The PPV contains a maximum of 150* test items designed 
to measure receptive vocabulary • For each item the stimulus 
word {noun or verb) is presented orally and the child is 
required to indicate which of 4 pictures corresponds to the 



*Only 100 given in Fall, 



word. Item? increase in difficulty and the child continues 
until he makes 6 errors out of 8 consecutive .items* His 
score is then the number of correctly answered items. 
Reliability of the PPV is in the .7 to .8 range. Since the 
test has effectively no upper limit for young children, 
coiling effects are not a problemr nor are there floor 
effects. Correlations with other tests in th»^ battery are 
generally high. The highest are .665 with the PSI and .537 
with the WRAT Recognizing Letters. •Although the PPV pro- 
bably taps general intelligence and language ability. Walker 
recommends that the test ^ be used only as a measure of 
passive vocabulary at this time. 

Wide Range Achievement Test (WRAT) 

Four of the WRAT subtests administered to the full 
sample in both the fall and spring were used in the 
analyses. The WRAT subtests measure specific academic 
skills; and it seemed reasonable to treat thein as separate 
measures. The PSI provides a good measure of general 
achievement. By looking at the various WRAT subtests 
individually we can obtain a more detailed profile of 
cognitive program effects. 
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; 1- WRAT Copylncy Mayk« g» btest (WRTC) 

, In a one minute time interval the child copies as 
i many of a series of 18 marks as he can. He is given credit 
j for t^e number judged by the tester to be copied correctly, 
i There are possible tester biases. Although our be^t 
i estimate of internal reliability is about .8, a severe floor 

effect, particularly in the fall/ renders this' figure less 
! impressive. Highest correlations are .551 with the PSI and 
: .508 with the ETS. Although it is not' clear exactly what 

useful .kills are related to being able to copy abstract 

markings accurately and quickly, the WRTC probably measures" 
motor coordination and a component of general school readiness, 

2. WRAT Recognizincy Letters fWRTRl 

The child is required to recognize and match letters. 
The tester points to a series of letters in a row, and the 
child picks out the matching letter from a different series. 
There are 10 items. Our reliability estimate is around .8, 
but there are a substantial number of children scoring 
0 and 10. Highest correlations are .537 with the PPV and 
.481 with the PSI. This test seems to measure the ability 
to.recognize letters and also, possibly the ability to : 
match shapes. 
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3. WHAT Naming. Letter (WRTN) 

The child is asked to name each of a series of 13 
letters. Reliability is estimated at around .95. There 
is, however, a severe floor effect, particularly in the fall. 
Highest correlations are .600 with the WRAT Reading Number 
subtest and .414 with the PSI. 

4 . VreAT Reading Number (WRTD) 

The child is asked to read aloud the numbers "3, 5, 6, 
17, 41." Reliability is estimated at about .6, but there 
is a floor effect in the fall. Also ther0 were almost 
no children in either fall or spring capable of identifying 
the number "41." Highest correlations are .600 with the 
WRTN and .508 with the PSI. 

Since these four WRAT subtests measure fairly specific 
skills, h^ve reasonable reliability, and were given in both 
fall and spring, we have decided to indlude them in the 
major analyses. The floor and ceiling effects will, however, 
raise problems in some of the analyses. / 

Illinois Te s t of Psycholinguistic Ability; Verbal % 
Expression subtest (ITPA ) 

The ITPA measures a child ^s ability to express him* 
self verbally. The ITPA is a diagnostic test, ahd its uso 
for evaluative purposes is experimental. The child is 
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<' * . 

f 

handed four familiar objects (ball, block, envelope, 
button) one at a time and asked by the tester to "tell me 
all about this." The score is the total number of distinct 
descriptors used by the child. Reliability is estimated 
to be between .6 and Highest correlations are .506 ^ 

v;ith the PSl and .487 with the PPV. . Although a child's 
ability to express himself would seem to be an important 
skill for later school success. Walker cautions that its 
usefulness in evaluation, is questionable "because of the 
large variance, overall low mean response rate, and test 
administration problems." 

EdUGjsttiei^al Testing Service Enumeration Test (ETS) 

The ETS as used in this study consists of 3 subtests 
designed to measure components of the processes involved 
in learning the concept of number. The first subtest 
(Counting) requires the child to count dots (for one point) 
and say how many there are (for one point) for each of 3 items. 
The second subtest (Touching) has 6 items which require the 
child to touch each of the dots on a page one time only. 
The third subtest (Same Number Matching) consists of 8 items 
requiring the child to find the picture out of three with 
the same number of objects as the stimulus picture. 
- , A total score (meocimum of 20) is found from the three 
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subtests, A fourth subtest (Same Order Matching) was 
originally included, but eliminated because it had low 
reliability and low correlations with the other subtests, 

J. 

Reliability is estimated at about ,75. Highest 
correlations are .584 with the PSI, and .SOS with the WRTC. 
Although possibly subject to tester effects, the ETS has 
good psychometric properties and attempts to measure 
aspects of a developmental process which probably bears on 
future school success. It is one of the stronger tests in 
our battery. 

As indicated in Table 1-3, several other tests were 
also administered to all or part of the HSPV sample. We 
shall briefly describe these tests and give our reasons for 
not including them in our analyses of program effects. 

The Eight-Block Sort Task (8**Block) examines maternal 
teaching style and mother-child interaction. The teat 
consists of two parts, one of which has a floor effe'ct 
o and the other a ceiling effect. Reliat)ilijty estimates are 
high, but correlations with other tests in the battery are 
low* The test was given in the spring at only 10 sites in 
9 models. Since replication across sites is important in 
assessing model effects, we decided not to use the 8-block 
in our analyses. 
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The Brown IDS Self^Concept Referents Test (IDS) 
attempts to measure a child's self-concept. The distri- 
bution of scores is negatively skewed and displays ceiling 
effects. There is evidence that for Head Start age children 
the test measures cognitive (especially vocabulary) skills 
as well as self-concept^ and that children try to select 
socially desirable responses rather than those applying to 
themselves. Walker concludes that "because of •••theoretical 
problems and the conflictir chnical findings^;,, the 
Brown not be used in this form i f^^^ure large-scale evalua- 
tions." 

The Classrqit>ni Behavior Inventory <CBI) assesses child 
behavior in three areas: task orientation, extraversion/ 
and hostility. For each, of 15 items, a^ rater (usually the 
teacher) rates the child on a seven point scale. Test- 
retest reliabilities are adequate, but Ihter-rater relia- 
bilities are moderate, and it is clear that different raters 
have different scales of reference, making cross classroom 
comparisons impossible. Since it appears impossible to 
obtain an absolute measure comparable across classrootns, 
there seems to be no way to use the CBI as an outcome measure* 



The Motor Inhibition Test (MI) attempts to measure the 



"35 



ability to inhibit movement to conform to task demlTnds. 

■i 

Only one of three parts (Tow Truck Task) was given in 1971-72. 

I . 
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Reliability does not appear adequate, and there are sig- 
nificant tester effects. Walker suggests that "the Truck 
subtest be dropped from future large-scale evaluations." 

Gumpgookies (GG) is designed to measure achievement 
motivation. Reliability estimates range from .7 to .9, 
The ft^prt form of the GG used in HSPV is experimental, and, 
considering that it was administered only in the spring, we 
felt it would be misleading to use it for evaluative purposes. 

TheNyelevant Redundant Cue Concept Acquisition Test , 
or "Zings and Poggles" (Z & P) tests a child" s, ability ^ 
to master a particular abstract concept. Reliability for 
Head Start age children is very low, and it seems that there 
is much guessing. The test may be good for older children, 
but i ft just too difficult for children this young. Further- 
more, it was given only in the spring. 

Besides the four WRAT subtests discussed above, one 
other was given in both fall and spring. The WRAT Dot 
Counting (WRTO) subtest requires the child tq count a 
series of 15 dots arranged in a row. The score is the 
highest number counted correctly. There aire both floor and 
ceiling effects (a substantial number of children scoring 
0 and 15). Moreover, ti ^ subtest consists of essentially 
one item. 



Four other WRAT subtests were given in the spring only. 
These are Spelling, Oral Arithmetic, Written Arithmetic, 
and Word Reading. All except the Oral Arithmetic subtest 
were Gleoily too difficult for Head Start children. Since 
the estimated reliability of the Oral Arithmetic was only 
.55 and we had no fall scores, we decided not to use it. 

We conclude our discussion. of tests with mention of 
the Hertz Ig-Blrcb scoring system, which was applied to the 
PSI in 1971-7?* This elaborate scoring system notes not 
only whether an item is answered correctly or incorrectly, 
but assesses the child's style of response to cognitive 
demands. The system is experimental and potentially quite 
informative, but more data is needed before its usefulness 
for evaluation can be determined, 

f 

Some of the data which we found not useful^ for our 
HSPV evaluation may prove useful in the study of Follow 
Through, and in particular in the study which will follow 
our HSPV sample into Follow Through. Accordingly, we have 
forwarded this data to Abt Associates^ We would also 
encourage others to study the various experimental tests 
in our battery, so that more refined instruments will be 
available for future studies. 
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We conclude this section with brief descriptions of 
other sources of information used in our analyses. A 
complete listing of the specific items used can be found 
in Appendix A. 

The Classroom Information Form (CIF) was our primary 
source of information on child background characteristics, 
such as age, sex, and ethnicity. These forms were filled 
oiit by teachers. 

The Parent Information Form (PIP) was administered to 
parents to elicit information about home environment, parent 
and child attitudes, and the extent of parent inyolvemen^: in 
Head Start and other activities. SincJe Control children 
were not in classrooms, their parents were given a modified 
version of the PIP which served also as the primary source of 
child background information. 

The Teacher Information Form (TIP) , filled out by 
teachers, requested information on teacher background, 
teaching experience, and attitudes towards the PV model 

(if any) with which they were working. 

i. * ' . 

In addition, several items of information were provided 
by sponsors and local Head Start directors. Of these, we 
utilize only the ratings (on 0 to 9 scale) of classrooms 
in terms of the degree of implementation. 
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Other Reports 

This report is one of a series being prepared under 
Grant § H 1926 from the Office of Child Development. In 
this section we discuss briefly relevant results from other 
reports in this series. 

Smith (1973) analyzed the 1970-71 cohort data* His 
outcomes consisted of three measures of cognitive achieve* 
menti one measure of general intelligence, and one measure 
of motor control The achievement measures were a ^4-item 
version of the Caldwell Preschool Inventory (PSI) , and the 
NYU Booklets 3D and 4A. The FBI is a test of general 
achievement in areas deemed necessary for later school 
success. The NYU books tap more specific cognitive skills. 
The Stanford-Binet IQ was used as a measure of intelligence. 
The Motor Inhibition test was used to measure a child's 
ability to inhibit physical activity in order to perform a 
specified task. The interested reader is referred to 
Walker's teport (1973) for more detail on these tests. 

On the basis of his analyses using these measures, 
Smith concluded that: 

« 

1. The Head Start experience substantially improved 
performance on all 5 outcome measures. 

2. There were no differences in effects, between the 

PV programs (taken together) and the NPV programs 

on any of the measures. 
« 

3. No model stood out as being overall more or less 
effective than the others. 
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,A more detailed breakdown of the. inter-model comparison 
results appears in Table 1-5, reproduced from Smith's 
report. Smith found two instances of outstanding model . 
effectiveness* . The High/Scope model was e^itraordinarily 
successful at boosting Stanford-Binet IQ scores. The ; 
Kansas model was highly effective 'in raising scores on the 
BooX 4A. , Only a few other effects are cited in this table, 
and' on the whole, it appears that there is not much 
difference in overall effectiveness of the various models. ^ 
in terms of the measures used. 

Featherstpne (1973) has s^ttempted to relate^ program 
effectiveness to child characteristics; that is, to detect 
model -by-(3hll<3 characteristic interactions. . Using the 
1969-70, and 1970-71 data and studying the PSI and Stanford- 
Binet only, she fin^ds no consistent, interpre table inter- 
actions Involving fixed child characteristics , such as . 
sex, ethnicity, and sol3io-economic status. She suggests 
that characteristics such as age, prior preschool experience 
and cognitive ^ style, which describe the child at a particula 
point in his development, may relate to relative model 

effectiveness. She concludes that "the strategy which 

.'>i 

works be^t for a child today is not necessarily the one 
which will be optimum next month or next year." 
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. (Reproduced from 5inith, 1973) 



$'uimrtary' of Planned Variation Model Effectiveness on Five 
» * ' ^ Outcoine Measures 



Zero (0) indicates model is^ of average effectiveness on 
. outcome measure. 

Minus (-) indicates mddel may be of below average effec- 
^ * tiveness, , 

i?lus (+) indicates model may^be o£ above averagS effec- 
,^ tiA^eness.i ' 

Double pins (++) indicates tnodel is probably highly 
. effective .v,^ t > 



v. 



Model 


Book 


■ Book 
4A . 


]>SI 


Stanford 
Binet 


Motor 
Inhibition 


Far West' ^ 
Laboratoxry 


0 

« 


0 


0 


0 


1 : 

■ 

0 


Arizona 




■ 0 


0 


0 


0 


Bank St.^ , 


0 


0 . 


b 




+ 


Univ. of 

OVpgon 


0 


+ . 


0 


0 


0 


Uitiv. of 

Kansas 


0 


++ 


0 


■ * 0 


+ 


High-^ . 
Scope 


+ 


. 0 


0 


++ 


0 • 


Univ. of 
Florida 




' 0 


.0 


" . 0 


0 " • 


Ebo 


0 




0 


- 0' 


^ 0' 


Univ. of 
Pittsbuj^qh 


• b 


+ 


0 




— i — — ^ ' — V. 


REC 

; , . . . <: 






0 




0 


En able rs 


0 




0 




+ 
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Lukas and Wohlleb (1973) have considered the process 
of program implementation • They explore the b^sic questions: 

1, How well are the models iirtplemented? 

2, What ^factors affect the process of implementation? 

It was originally assumed that models were well- 
defined carefully specified educational packages which, 
given sufficient time, could be replicated completely 
at any chosen sTite. Lukas and Wohlleb have found that 'the 
implementation process is much more complex, involving a 
large number of people (sponsors, local administrators, 
parents, teachers, teacher-aides) with varying, and sorne- 
times conflicting interests, goals, and philosophies. 
Moreover, the models themselves are not that well expli- 
cated by the sponsors. We cannot be sure exactly what 
treatment is being received by children in a given class- 
room simply by knowing ^the model name. Classrooms with 
the same model may vary considerably. 



Summary and Lbok Ahead 

In this chapter we have tried to give the reader a 
picture of the HSPV experiment. We began with, the back- 
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ground of ^the study, showing how it evolved out of earlier 
Head Start programs and evaluations, and discussed its 
relation to the Follow Through program. We then set out 
the three major questions we hope to answer: 

1. To what extent does a Head Start experience ac- 
celerate the rate at which disadvantaged pre- 
schoolers acquire cognitive skills? 

2. Are the Planned Variation models, simply by virtue 
of sponsorship, more effective than ordinary non- 
sponsored Head Start programs? 

3. Are some PV models particularily: effective at im- 
parting certain skills? 

We then provided brief descriptions of the 11 PV models 
to be studied. It was noted that an important dimension 
in describing these models is the extent to which the ac- 
quisition of academic skills is stressed. We noted that 
at least three of the models are consciously concerned with 
the development of specific academic skills useful in the 
early school years. These are the Oregon, Kansas, and Pitts 
burgh • models • We shall refer to these throughout this 
report as the "academic" models • As an important sub- 
question of the thir4 of our major questions, we shall ask 
whether the academic models are overall especially effect- 
ive . 
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Following the model descriptions, we set out the 
basic design for the study. We defined, ja site as a group 
of children in a particular location undergoing a partic- 
ular kind of preschool experience. There are 28 PV sites > 
each with one of our 11 models, 12 ordinary non-«ponsored 
(NPV) sites, and 3 Control sites, with ro preschool pro- 
gram. We also decided for convenience in presenting re- 
sults throughout this report to refer to the NPV children 
taken together and the Control children as program groups. 

We then discussed the data collected. A brief des- 
cription of each instrument was given. For each test we 
gave our reason lor including or not including it in our 
analyses. V^e found that only eight tests were suitable 
for program evaluation. Thtfee are the Preschool Inventory 
(PSI) r Peabody Picture Vocabulary Test (PPV) , four subtests 
o£ the Wide Range Achievement Test (WRAT) , the Illinois 
Test of P?ycholii)guistic Ability Verbal Expression Subtest 
(ITPA) and the Educational Testing Service Envuneration Test 
(ETS) . Unfortunately, these tests all measure skills in 
the cognitive domain, frustrating the hope that information 
on^socio-emotlohal development might also be obtained. 

Finally, we presented brief summaries of relevant 
findings from other reports in this series. With all this 
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as background/ we are now ready to look at the data. 

The remainder of this report consists of 8 additional 
chapters. Chapter li contains a descriptive presentation 
of much of the data. We present background character- 
istics of children and teachers r and summaries of the dis- 
tributions of fall and spring test scores for modeld and 
sites. Chapter III JLs devoted to general methodological 
considerations ♦ We discuss limitations placed on the 
analysis by the design and the tests themselves, And our 
general approach to the problem of evaluating educational 
programs* Chapters IV through VII consist of four differ- 
ent analyses of the data, each with certain strengths 
and weaknesses. Through this variety of approaches, we 
hope to gain answers to the major questions mentioned 
above. In Chapter VIII we consider what evidence we have 
relating to the question of interactions between individ- 
ual child characteristics and program effects. Finally, 
in Chapter IX we summarize the results of the various 
analyses and present our conclusions. 
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Chapter II 
Desorlptlve Presentation of the Data 

Introduction 

In this chapter we present svunmaries of the. data In 
order to give a general idea of the sample in teriaef of back- 
ground characteristics and distributions of test scores. 
The reader Is forewarned that th^ chapter is largely a rather 
dry conq[>ilation of facts, inbluded primarily for the sake 
of completeness and future reference* 

Note that there is no one analysis saii|)le to whi6h we 
can always refer. The samples suitable for the different 
analyses described in this report may differ slightly . Not 
all the same information has been collected on each child, . 
and the minimal data requirements -for the various analyses 
differ.^ We shall always make clear the criteria used in 
selecting an analysis saitqple. In general, the data collection 
was well done, and there is little missing data on key 
variables. 

Child Background Characteristics 

Tables II-l through II-3 present some iit5>ortant back-^ 
ground characteristics by model and site for a sample of 
3,361 children. This represents all children whose sex is 
known, who are either Black, White, Mexican American or 
./ Puerto Ridan, emd who have a valid fall or spring test score 
on at least one test. A test score was considered valid if 
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the tester indicated that the test was conpleted^ Although 
reasons for failure to coirplete a test were noted by the 
tedter, no incortplete test results were used in our analyses i 
regardless of the reason. 

Looking first at Table II-1| we see that there is an 
approximately even split of boys and girls in all iftodels. 
The ethnic coirpositions of the sites vary greatly. The 
Pittsburgh iTK>del, for exairple^ contains only Whites > while 
the Bank Street inodel is 83.3% Black. The Mexican Americans 
are distributed throughout seven of the models, with 
Oregon and REC containing by far the largest nun4>eti While ^ 
most children come from homes where English is spoken/ 
Oregon has 41.9% where this is not the case. TJie percentage 
of children with some prior preschool experience varies 
considerably. EDC (45.3) and Bank Street {40.9) are high, 
and Kansas (7.6) is low. The average number of years of 
mother's education varies from 9.4 for Florida to 11.1 in 
both Par West and Pittsburgh. The mean age (on October 1, 
1971) varies from 53.4 months for Pittsburgh to 64.7 for 
Oregon. The Control children are a bit younger than the 
Head Start children! averaging 49.6 months. Household size 
varies little, from an average of 5.0 in Far West to 5.9 
in Kansas. Mean family income also shows little variation! 
ranging from $3000 per year in Florida to $4,410 in 
Pittsburgh. 

Tables II-2 'and II-3 give the same background infor- 
mation broken out by sites. For all varicibles, with the 
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possible exception of sebc emcl household size, there, is 
considerable site-to-site variation within models. There 
is only one site with a mean age between ^7 and 63 months. 
Essentially there are two distinct types of sites, those 
in which children were to enter kindergarten following 
Head Start, and those in which they were to enter first 
grade directly. Smith (1973) suggests that iphildr'en in 
"entering-first" sites may undergo systematically different 
experiences from those in "entering-kindergarten" sites .j 
This is an interestfing hypothesis. Unfortunately entering 
level is severely confounded in our design with age, 
region, and model, all of which would have to be controlled 
in order to tease out the entering-level effect. Thus we 
have not taken any explicit account of entering level in 
our analyses. We do, of courset, recognize age as a poten- 
tially important influence on measured outcomes. 

Teacher Background Characteristics 

In Table II-4 we present background information by 
model for all teachers. Nearly all the teachers are women 
and are either Black or White. The percentage of Black 
teachers ranges from 0 in Pittsburgh and REC to 66.7 in 
EDC. The percentage of teachers living in a neighborhood 
similar to that of the children they teach ranges frpm 0 
in REC to 88 in High/Scope. The percentage of teachers 
certified ranges from 0 in Oregon and Kansas to 50 in REC. 
Mean teacher age ranges from 26.8 years in Far West and 




Table 11-4 
Teacher Background Characteristics * 



•Program 


% 

Female 


% 

Black 


ft Q A1Y1A 

Neighborhood 


Certified 


ncan 
Age^ 


Mean 
Yrs. Ed, 


n 


Par West 


100 


5.' 6 


16 . 7 


33.3 


26.8 


16.0 


18 


Arizona 


100 


37.5 


■ 50.0 


37.5 


36.5 


15 . 3 


16 


Bank St. 


100 


60.7 


46.7 


36.7 • 


38.0 


15.2 


- 30 


Oregon 


100 


22.2 


• 66.7 


.0 


37.4 


13.6 


9 


Kansas 


' 100 


25.0 


81.3 


0 ■ 


. 7 ' 
<aQ k f 






H ^cih/SoAiM 


100 

«i> V w 


16 . 0 




« 0 . u 






* 


Florida 


100 


50.0 


22.2 


t - 


04 . O 






EDC 


100 


66.7 


77.8 


33.3 


34.9 


14.9 


9 


Pitta- 
burgh 


100 


0 


44.4 


33.3 


34.9 


16.0 


9 


REC 


50 


0 


0 


50.0 


26.8 


16.5 


4 


Enablers 


85.7 


19.0 


12.0 


. 28.6 


41.5 


14.8 


21* 


NPV 


96.2 


25.0 


38.5 


45.3 


37.7 


15.2 


52 


Total 


96.9 


29.3 


55.5 


33.3 


36.7 


15.1 


227 



*Thi8 table is based on all teachers, including isome HSPV 
classes in whiOh no testing was done. j 
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REC to 41.5 ^In the Enablers. Meam years of education varies 
from 13.6 in Oregon , to 16 ;5' in-* REC » , . 

J* ■ ■ 

Outcome" Measures 

^ Tajolee II-5 through I It 2 8 present summary statistics 

for the distributions of fall and spring test scores for 

models and sites. For each model or site and each test^ we 

present the mean/ median ^ lover quartile^ upper quart ile^ 

standard deviation, and sample size. The sample used for 

each test consists of all children with valid fall and 

spring scores on that test. 

It is evident that there are substantial di.fferences 
among models and among sites within models on background, 
characteristics and fall test scores. Our ability to make 
fair comparisons among programs will depend on our ability 
to tcike account of and adjust for these pre-program differ- 
ences. Since experimental equalization has apparently 
fail.ed/ we must rely on statistical techniques. We shall 
discuss how and to what extent this is feasible in the 
following chapter. 

Many researchers feel comfortable in describing gains 
or effects in terms of steindard deviations. As a roagh rule 
of thumb, one-*half a standard deviation is sometimes taken 
a& a criterion for educational significance. For convenient 
reference, we list in Table 11-29 the stemdard deviations 
based on the Entire fall seorple of children in Head Start 
programs. 
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Table li-29 

Means and Standard DevlaClons for Entire Head 
Start Sample in Pall 1971* 



Test 


Mean 


1 Standard Deviation 


h 


PSI 


14,59 


6.16 


2972 


PPV 


31.53 


» 13.26 


2996 


WRTC 


1.92 


2.67 


2980 


WRTR 


6.55 


3.21 ■ 


2980 


WRTN 


1.20 


2.63 


2980 


WRTD 


.61 


1.10 


2980 

■ 


ITPA 


11.28 


5.16 


1204 - 


ETS 


11.65 


4.84 


1129 . 



\ 

all children on whom age information 



*This sairple i^^ eludes 
was available. 
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Chapter III 
* Gene3g!fal Methodological Issues 

Introduction 

In Chapter I we gave an overall picture of the HSPV 
study. In Chapter II we took a first look at the data 
collected. Before going on to the analyses carried out, 
we thought the perspective provided by a general discussion 
of methodological issues would be valuable. We begin with 
a discussion of the major difficulties resultingf from the 
study design. We then discuss the bag of statistical 
tricks usually used in attempts to overcome such diffi- 
culties. We at first ignore the thorny problem of measure- 
ment error, and later discuss its effects on the various 
statistical techniques. Finally, we discuss the general 
'research strategy we have adopted. 

Design Prol^lems 

Undoubtedly the most serious design problem in this 
study is the lack of randomization . If a group of experi- 
mental units is divided randomly into two or more groups, 
then providing the groups are sufficiently large, there is 
only a small probability that they differ significantly on 
any given variable, measured or unmeasured. Of course. 



76. 



we can neverx be sure that the groups are equivalent with 
respect to all variables, but randomization is our best 
protection that there are no relevant group differences • 
If allocation to treatment .groups is random we can be 
fairly confident that comparisons among group outcomes 
are unbiased even if no explicit account of pre-treatment\ 
variables is taken ♦ We may still wish to use pre-treatment 
information to increase the precision of our comparisons, 
but with random allocation this information is more / 
a luxury than a necessity. 

In the HSPV study we would ideally have liked the 
group of children assigned to each model to be a represen- 
tative sample of potential Head Start children. Since we 
cannot transfer children around the country at will/ the 
smallest unit in which children can be assigned to models 
is the site. Thus, for purposes of model comparisons, 
randomization would have to be employed in the assignment 
of sites to models. Since there may well be systematic 
differences among the pools of children at different sites, 
it would be necessary to have several sites assigned to 
each model. With only 2 or 3 sites per model, substantial 
differences in the children assigned to various models 
would be likely even if randomization were employed. 

Since we have so few sites and assignment was not 



random I we cannot assume that the children assigned to 
various treatments are sufficiently alike to allow direct 
comparisons. A quick glance through the tables in the 
previous chapter reveals that, in terms of at least sonie 
variables undoubtedly associated with academic performance 
there are some obvious and pronounced dif f erences» . There 
is clearly variation among models and among sites within 
models in ethnicity, age, mother's education, prior pre^ 
school experience, and, most important, fall test scores. 
It will be necessary to in some fashion take account of 
these differences in our analyses, V?e will never know for 
certain, of course, whether our adjustments suffice to 
provide fair comparisons of program effects, but we hope 
to make a convincing case for their adequacy. ; 

/The unbalanced nature of the design in terms of back- 
ground characteristics causes particular problems for the 
measurement of interaction effects* If we wish to relate 
program effectiveness to various background variables, 
we would like to have the distribution of these variables 
similar in the various programs, and representative of the 
full range of variation in the Head Start population. As 
an extreme example, suppose we are t^rying to relate model 
effects to ethnicity. Since the Pittsburgh model has only 
White children assigned to it, we have no data to address 
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the question of how its effect varies for different ethnic 
groups • 

Of particular concern in connection with the lack of 
random allocation is whether the Control children differ 
in any systematic way from the Head Start children. We 
wish to use the Control results to estimate what would happen 
to a potential Head Start child if not enrolled in a pre- 
school progtam. Since the selection mechanism is of ne- 
cessity different from that for Head Start, there may be 
important differences. For example, we note with some 
•concern the fact that the Control children t^nd to be 
younger than the Head Start children. In fact, there are 
a substantial number of very young (less than 4 years old) 

children in the Control sample. These may well be waiting- 

» 

list ^children deemed not yet old enough for Head Start. The 
fact that their mothers are applying so early may irldicate 
that there is something special about such children. We 
really don't know, but w^ cannot be sure that Control 
children are suf fibiently similar to Head Start children to 
be used to. measure absolute effects of Head Start. Our 
attitude in general will be to compare Controls with Head 
Start children, but\to be circumspect in interpreting the 
results. , ^ ^ ' ' 

ERIC 



approaches impossible to justify unless we have strong 
evidejnce that the groups are a pjMori equivalent. 

The simplest approach which takes some account bf- ' 
pre-treatment differences is to conapare average gain 
scor^es . A gain score i6 simply the difference between 
post-test . and pre-test gcoj^es. By \i^ing gain scores we 
implicitly assume a mathematical model which states that 
on the average, i,f treatment effects were equal, the post- 
score would equal the pre-score plus sojtie constant. This 
is a very restrictive model. It says, fdr example, that 
given the same program, if children with a fall scor6'pf 
10 on the PSI obtain 17 qn the" average in the spring, then-* 
children with a score of 20 wilj. or) the average obtain 27. 
It is rare that a test is calibrated so as to make such an 
assumption reasonable. 

Gain scores also, of course, take no account qf back*- 
ground variables other than the pre-test. Analyses using 
gain scores as outcomes and adjusting f^r other variables 
are possible. It seems, however, more natural to use 
approaches which 'are more flexible in .%h^ way pre-tests and 
other variables are used together to adjust post-test scores 
Three such approaches are how considered* In describing 
them we si .11 refer to all variables used for adjustment of 
the post-test scores; as ''cbvariates • " 
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A simple way of comparing progrjims is to cross- 
classify subjects on the basis of the covariates and , 

' i ■ ■ 

directly compare. tJje average scores for subjects in the 

Some class in the two treatment groups • Suppose, for 

• f 
example^ that we stratify children by ethnicity, a^e, and 

pre-tjBSt scoye/ Then Whites between 50 and 55 months of - 
age with fall scores between lO' and 15 in the two progtams 
could be directly compared. Such comparisons will be 
unbiased with respect to the covariates used in the cross- 
classification. The'^ approach is simple and the results 
easily understandable / but it generates a mass of information 
wh\ch may be difficult to use, , v 

Suppose we can meaningfully specify a reference popu- 
lation (in terms of covariate distribution) which is of 
interest (often the entire sample is used for this purpose) . 
Then by appropriately weighting the subgroup means for the 
two treatments, we can estimate the average outcome score 
which would result from applying each treatment, to the. 
reference population. This technique, is known as direct 
standardization . For example , ^suppose we sub-divide 
according to sex and pre-score, and obtain the hypothetical 
results illustrated* in Figure III-l. Note that the overall 
post-sc^e me.an for Model A is 19.3 and for B is 18,2, 
.even though the mean for each sub-class is at least as high 

\ 
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for B as for A. Now suppose we apply the observed sub- 
group means to a standardized population with 25% in each 
of tue four sub-classes. The standardized mean for Model 
A is now 19.0 and for B 19. 5, These numbers present a \ 
fairer comparison, in that effects resulting from imbalances 
in sex and prior pre-scoreis have been removed. 

The major difficulty with sub-classification approaches r 
including dire<?t standardization, is that in, order to 
exercise greater control over biases, we must sub-divide 
the sample more finely. This leads tSo fewer obse^ations 
per sub-group and less precise estjfmation. • 

Possibly the most popular approach at the present time 

is /the analysis of covarlance (ANCOyA) • It Is based on the 

assumption that the expected value of an% individual ' s post- 

test score is a linear function of a set of ipe^^urable^ 

variables. Thes^ may be continuous variables, dummy variables 

representing membership in various\classif icatory groupings, 

or variables representing interaocions amqng directly 
* » 

measured variables or transformations of them. Thus the 
expected outcome can in principle 'be expressed as a function 
Of d'Jmmy variables corresponding to the programs we wish to 



*A dummy variable is one which assumes a convei\tional value • 
(usually 1) for all individuals with some specified property 
and anbther value (usually 0) for those without the property. 



Figure III-l 
Illustration of Direct Standardization* 



Model A 



Model B 



re-score i 12 



> 12 



tan 



Male 


Female 


15.0 


18.0 


(10) ■ 


(15) 






23,0 


20.0 


(15) 


(16) 



19.3 (=i^ X 15 + X 18 



50 50 



andardized Mean 19.0 



Male 


Female 










15.0 


19.0 






■■ 
















■J 

24.0 


20.0 










(5) 


(15) 

^ .... 


















i 




8 18.2 




15 


+ 






20) 




24 


+ 


if X 20) 




18 19.5 


(» .25 


X 


15 


+ .25 X 


19 


20) . 


+ .25 


X 


24 


+ .25 X 


20) 



Reference Population 



1. 


Male 


Female 


* 


' ) 

e- score' < 12 


.25 


.25 


> 


>12 


.25 


.25 




V 









or each sub-class/* the top number represents the correspording mean 
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compare as well as a variety of covariates, We can 



then estimate the relative effects of the treatments 
after "adjustment" for covariate differences^ estimate 
the proportion of total post-test variance explained by 
program differences over and above that explained by the 
covariates r and test the significance of adjusted program 
differences. If the ANCOVA model is approximately correct, 
it is a powerful and flexible instrument for group compari- 
sons. We shall discuss' the theory underlying ANCOVA in more 
detail in Chapter VI. 

A common approach which avoids the necessity to specify 
a particular mathematical form for the relationship* between 
outcomes and covariates is matching . In its simplest form 
matching involves finding pairs of subjects in different 
treatment groups with effectively Identical covariate , 
values. Any difference between post-test scores of the 
members of such pairs cannot be attributed to differences 
on the covariates* Each pair provides an unbiased comparison ' 
between two treatments and by averaging we obtain an estimate of 

program difference. Since in practice we ^can almost never 
find exact , matches, the efficiency of the matchi;rig procedure 
depends on our^Aability to find "good" matches. This can ' 
be a serious problem. The most up-to-date .theory on this T 



subject is by Rubin (1973) . 
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If the assumptions of ANCOVA are approximately correcti 
it uses the data much more efficiently than matching. 
Matching, on the other hand, has the advantage of robust- 
ness. That is, it requires almost no assumptions on the form 
of the relationship between covariates and post-test scored 
to^ be valid • Combinations of matching and ANCOVA tech- 
niques are also possible^ The interested reader is referred 
to Rubin (1973) and Smith (1973). 

* - ■ . r 

Effects of Measurement Error on Standard Analyses 

Up to this point in discussing standard approaches 
to the problem of accounting for initial differences 
between treatment groups, \ :jve not considered the fact 
that what we measure may be only an approximation t'o a 
true variable of interest. Classical measurement theory 
(see Lord and Novick, 1968) defines the reliability of a 
variable as the percentage of its variance (over some 
specified fJopulation) attributable to variation in the true 
score. This notion is meaningful if we assume that the 
observed, score is the sum of true and error components, 
where' the error has mean 0 and is uncorrelated with the , 
true score. - . 

; inf general, we ai/e^uch more concerned , about the 
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reliabllijby of c -^^ lates than ot the outcome measure. 
Under the classical measurement model, at least/ the random 
noise introduced by errors in the post-test score tends 
to make our inferences less precise, but does not intro- 
duce systematic biases. Error in the covariates, on the 
other hand, causqg serious problems, in the standardization 
approach, for example, we try to create relatively homogenous 
subclasses, if our classification is on the basis of 
variables measured with error, the subclasses may be less 
homogeneous than we believe. Substantial misclassif ication 
may result in serious biases. 

Effects of measurement errors on ANCOVA can be equally 

devastating. Suppose that an ANCOVA model using the 

.. - , > • 

(unavailable) tifue covariate scores would accurately describe 

the situation, in the econometric literature, the" equations 

relating expected outcomes to true covariates are known 

as . structural eqvations . If we use our observable variables 

to fit a linear model, the resulting parameter estimates 

turn out to be biased estimates of the structural parameters • 

A biased treatment comparison will result, with the nature 

,of the bias depending upon the nature of the measurement 

error. ' ) 

In the one covariate situation, Lc/rd (1960) and Porter 

(1971) assume the classical measurement model and suggest 
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techniques for obtaining fair ccanparisons . Other recent 
work (De Gracie and Fuller, 1972i Stroud, 1972) has addressed th 
question of "correcting" linear models under various 
assumptions about the errors. All these approaches are 
mathematically complex, and it is not clear at this point 
which, if anyr are really suitable for educational quasi- 
fexperiBwants . 

Instead of assuming the existence of a true mo^el 
involving structural equations, we can decide to deal 
only with observables, and to build the best model we can^ 
Under this approach the only way to insure against possible 
^ biases is to find covariates with high reliability, as 
well as a strong relation to outcomes* This approach has 
the advantage of simplicity. A sophisticated statistical 
correction which can be implemented only crudely tnay .well 
be more misleading than no correction at all. 

In matching, if there are errors in the covariates, 
we will be matching on the basis of possibly incorrect 
values. A true match would occur if the members of a 
matched pair ha^ identical true scores. Under the classi- 
cal TOpasurement model, an individual's true score on a 
variable on the average somewhere between his observed 
score and the mean for the populacion from which he is , 
selected. Thus, two individuals may have the same observed 
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score on a variable but true scores which diffej^. If this 
variable has an effect on outcome scores, this (unobserved) 
difference may affect the observed post-test difference. 

9 

General Analysis Approach ^ ] 

In this section we discuss this general principles 
guiding our analysis plan. If we were dealing with a 
carefully designed randomized experiment, the analysis 
strategy would derive naturally from the design. Unfor- 
tunately, as explained above, we do not have such a situation. 
Campbell and Erlebacher (1970) have argued that the problemi? 
caused by lack of randomization combined with imperfect 
covariate reliability are virtually insurmountable. 
They seem to agree with Lord (1967) that "no logical or 
statistical procedure can be counted on to make proper 
. allowances for uncontrolled pro-existing differences 
between groups." CertalAly there is no substitute for a 
randomized experiment/ but we feel that the "randomization 
or bust" reaction is overstated. By applying several 
alternative analysis strategies to Our data, we feel it 
will be possible to obtain a fair assessment of the relative 
impacts of various preschool experiences. Each of our 
\ analyses will have its own strengths and weaknesses in terms 
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of the ability to detect real ^'effects." Each depends 
tor its validity on a set of assumptions* These assump- 
tions correspond to certain Mathematical models which 

, describe aspects of children's learning processes. We 

* ■ , 

shall try to make the assumptions and corresponding models 
as explicit as possible, so that the reader can judge for 
himself the validity of the various analyses. At the 
very least, it should be possible to make conditional 
inf 6f en^es ~thr~fsm"^ „ ^ la tTu^ hypc^theBiB 

B is supported by the data." Moreover, the pattern of 
results from the whole set of analyses will hopefully give 
us more insight than would be possible with a single 
analysis strategy. In particular, for any one analysis, 
•it is quite possible that a mathematical artifact will 
pass for a real effect. It is far less likely that an 
effect which shows up in several analyses based on different 
mathematical models is an artifact. 

Use of multipl e analyses , then is one of the principles 
guiding our analysis plan. Another prii>ciple is conservatism , 
Like Smith (1973), we intend to be cautious and qonservative 
in declaring differences among models. We would rather risk 
missing a marginally significant difference than declare 
a difference significant when it really is not. There are 
two main reasons for this policy. First, on the basis of 
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both our own Intuition and Smith's results, our expectation 
is that relative to the no-preschool condition, Head Start 
programs are quite homogeneous. Second, because of the 
implementation problems alluded to in Chapter Ir we can 
never be sure that apparent effects are really the result 
of programs. In comparing models, we assume a strong common cdm- 
ponent to the experiences of children in a given model. 
In reality, a model is a complex combination of the 
sponsor's original conception and many factors which 
affect its implementation in a particular site, or class. 
In many ways it is preferable to consider the model-site 
combination as the treatment. Thus we shall look for 
effects which are consistent not only across analyses, 
but also across sites within models. Unfortunately, one 
or our models (REC) was implemented in 1971-72 at only 
one site (Kansas City) . Althoucfh the data from this site 
will be analyzed arid results presented, we shall not draw 
any conclusions about the REC model's relative effectiveness. 

A third principle underlying our analyses is an 
emphasis on estimation of effects rather than formal 
testing of hypotheses. We feel that the resulting infor- 
mation is more useful. Given the large sample sizes with 
which we are dealing, statistical significance may be 
achieved by an effect which is educationally insignificant.- 
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It is of course important to know whether an apparent 
effect could be the result of chance variation. In the 
setting of a complex cjuasi-experiment with multiple 
analyses, however, this is no easy matter. We can, 
in effect, use our data to test many hypotheses, using tests 
that are often not indepehdent* Thus, even if the mathe- 
matical model underlying a testing procedure is correct, 
the formal significance level is illusory. There really 
isTno^true^^g^ "levels Sigin^flcalice'^Tevels^ 

therefore be used as suggestive indicators rather than 
formal certifications of real efxects. 

In the next four chapters we attempt to implement 
the analysis plan discussed in this section. In Chapter IV 
we describe a '^ranking analysis" which is intuitive and 
^ valid under minimal asstjmptions, though possibly conser- 

vative in detecting effects. In Chapter V we present a 

)' ' , , . ^ 

"j['esidual analysis" which attempts to partition the 
observed gains for different models into a part attrl- 
/ bu table to natural maturation and a residual attributab^^e 
to program effects. Chapter VI describes a conventional 
analysis of covariance. Chapter VII discusses a "resistant 
analysis" less sensitive to certain departures from the 
assumptions on whi^ph ANCOVA is based. The analyses 
described in Chaptex^^IV, V and VII have not to our knowledge 
been used before in educational evaluation. We hope others 
may find these approaches useful additions to the standard ; 
.^'*^ag of tricks" described above • 
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RANKING ANALYSIS ' 

Theory of Ranking Analysis 

The ranking analysis is simple^ intuitively appealing, 
and valid under minimal assumptions. It does not, however , 
provide a precise numerical measure of program effective- 
ness and may be conservative in detecting model dif fei^ences • 

The other techniques described in this report provide more 

• • • 

precise measurements at the cost of more stringent assumptions 

The ranking analysis is based on the idea that If / 
two individuals are exposed to equally effective progf^s, 
the relative order of the individuals in terms of their 
true scores on a measure should remain the Scime f rom pre- ' 
to post-test* Suppose that each individual hai ^^true* 
score T which is an. increasing function of time. Suppose 
further that we can measure the true scores of two individuals 
(say individual 1 and individual 2) in different programs 
at two points in time {t^ arid tj) . The relative order of \ 
these two individuals may remain the sa/ne (see Figure IV-la) 
or be reversed (see Figure IV-lb) . 

In order to judge whether individual 1 has gained 
relative to 2 in situation (a), we would need some notion of 
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the kind of growth patterns we would expect under the "hull 
hypothesis'^ of no program differences • Without this know^ 
ledge we can make no inferences from (a) • 

In situation (b) , on the other hand, it is clear that v 
1 has gained relative to 2, since he started below but 
ended above. Noto t-.hat this argument is plausible regard-* 
^ less of any differences in the individual?' background 
characteristics. Individual 1 may be below 2 at time 
for any number of reasons (e.g. younger, mother *s education 

4 

less, less prior preschool experience). It seems reasonable 
that whatever factors cause him to be behind at will 
continue to operate so as to keep him behind at t2 r unless 
ho is exposed to a rtore effective program. If prograuns are 
^equally effective, the developmental process way be such as 
to change the difference between scores over time, but it 
seems unlikely that the growth curves will cross. 

Basically, then, we assume that situations like <b) 
are evidence for differential program effects. Of course, 
there may be meaningful effects with (a), but we must rely 
on other analyses to detect them. 

Let us now extend the above argument to the ciase of 
several models* and several individuals per model. Suppose we rank 

_ ■ ■ - _ ' _ / _ _ ■ o 

♦Recall that ve consider the MPV sites pooled together as a 
model. 
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order the entire^ sample for both the pre-test and post- 
test |i Let the individual with the highest score have rank 
1, the second highest rank 2, and so on* If ^ a particular 
mod^l Is effective (ineffective), then the ranks of the 
individuals in that model should tend to decrease (increase) 
frbm pre-test to post-test. Looking at the average rank 
on pre- and post-tests would be a simple way of assessing 
program effectiveness. 

There is only one possible flaw in this argument. 
Suppose there are strong interactions between the relative 
effectiveness of the programs and some characteristic of 
individuals. Thcin the individual differences may be con- 
founded with program effectiveness. For example^, suppose 
program A is highly effective for boys but not for girls, 
and B is highly effective for girls but not boys. Then 
the observed relative effectiveness of the two programs 
will depend on the proportions of boys and girls in the 
two programs. 

Even assuming thete are no interactions between 
program effects and child characteristics, there is still 
another serious problem in implementing the above approach « 
We know that the reliabilities of our measures range from 
about .6 to .85» Thus, the observed rankings of individuals 
would be determined in part by random error variation. 
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To explore mathematicAl models to describe this situation 
would be a useful exercise, and might lead to statistical 
procedures which could be used to test the significance 
of observed rank changes. At present, however, no such 
procedure is available, and we have therefore taken a some- 
what different tack in dealing with the problem of less 
than perfect reliability. 

Often the reliability of certain group means is high, 
even when the reliability for individuals is quite low. 
By using the classroom, or site, as our unit of analysis 
instead of the individual, we may obtain higher reliability 
at the cost of fewer degrees of freedom. Since there is no 
sampling theory, to provide significance tests, having a 
large number of degrees of freedom is not especially useful* 
We felt that our primary goal must be to achieve virtually 
perfect reliability. Site means met this requirement 
reasonably well. 

The reliability of site means depends oji the reliability^ 
of the test for individual s, the number of individuals per 
site/ and the percentage of total individual variance which 
lies between sites. Thus measures Will vary in terms of 
thie reliability of site means, and the closer the relia- 
bility is to 1.0, the more confidence we will have in the 
analysis. For more detail on the way site mean reliabilities 



were estimated, the reader Is referred to Appendix B* 

Using site means still leaves us with the problem 
that if model effectiveness is related to child character- 
istidS/site variation in terms of such characteristics 
may come to be confounded with program effects. To get 
a partial handle on this problem, we performed the ranking 
analysis not only for the whole sample^ but also for 
varlG^s sub-samples. We performed separate parallel 
analyses for Blacks only, Whites only, Mexican-Americans 
only, males, females, children with prior preschool 
experience and children with no prior preschool experience. 
If model effectiveness were strongly related to background 
characteristics, we would expect Substantial differences 
in the results of these various analyses. By and Isirge, 
the resultfif we^e quite consistent, increafiting our con- 
fidence in the validity of the ranking procedure applied 
to the whole sample . 

In declaring a model particularly effective (or in- 
effective) on the basis of this analysis, we will take into 
account both the amount of the improvement in site ranks 
from fall to spring and the consistency across sites within 
a model. Note that since we are banking on virtually 
perfect site-mean reliability, we do hot, in theory , expect 
any "random" component to the changes in rank. However, 
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since reliabilities are not actually perfect and there may 
be some interaction effects ^ We must expect a little 
variation not attributable to programs* Moreover, sites 
starting out low have more opportunity to improve their 
position "by chance," and sites starting out high have 
more opportunity to lose ground. Also, since the distri- 
bution of site mean scores is probably most concentrated 
near the center of the distribution, we would expect more 
change for sites hearer the center • It is difficult to 
weigh these factors • Our judgments are subjective ^ and the 
reader is encouraged to draw his own conclusions from 
Tables IV- 1 through IV-9. 

We conclude this section with a suaeinct reiteration 
of the two assumptions on which the validity of the 
ranking analysis depends. 

Assumption 1 ; Developmental growth curves (in terras 
of true scores) for two site means will not cross during 
the period of program exposure unless the programs at the 
two sites differ in effectiveness* 

Assumption 2 : Site means have high enough reliability 
that the ranking of observed means is virtually identical 
to that of true score means. 
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TABLE IV- 3 

RESULTS OF WRTC RANKING ANALYSIS 

1 « Highest Site Mean 
40 » Lowest Site Mean • 
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RE SULTS OF WRTN RAKING ANALYSIS 

1 » Highest Site Mean 
40 •> Lowest Site Mean . 
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TABLE IV-6 

RESULTS OF WRTD RANKING AIlALYSiS 

1 " Highest Site Mean 
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^ « f : — 

• •• NPV 


SITE 


■ FALL • 


SPRING 


FALL 


SPRING 


T.04 

<t ' Att 

2^,13 


32 
• 21 
2 


: 6 

22 




*^ \ ■ 


-3.05 
3.08 
3.09 
3.16 


1 

26 
27 


1 

1 

\ 25 


36 . 

1 , 




5.10 
5.11 
5.12 ^ 


35 
- 37 


10 

Ho --^ 

39 


J 




7.11 
7.14 


13 

11 - 


11 
9 


J] 

8 • 


. 2 
32 


8.04 
8.08 


— S;; 

16 


-20 ' 

' 30.^ - ,. 


• 31 
17 


37 
26 


9.02"^ 

7 • U 4 

9.06 


22 


2$ . 
21 
30.5 




'12 


10.02 
10.07 
10. 10 


3 
25 

29 . 


7 
16 

23 


'6 
' 23 


3 
33 
15 


il.06 
"11.08 


18 
10 


8 


33 

7. 

r 


17 
•5 


12.03 
12.04 


38 
28 


18- 

19 . 






20.01 


12 


27 ■ 


20 


^ 13 


27.02 
27.03 

3^ 
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9 
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38 

3^^ 
4 

35 
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1 


1 
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TABLE IV-8 

RESULTS ETS RWKHjG ANAL YSIS 

I ■ Highest Site HeAn' 
40 «* . Lowest Site Mean 





PV 




SITE 


PALL 


SPRING 


FALL 


SPRING 


2,04 
2; 09 
2.13 


19 


14 

- 27 
15 




: . . — i^' 


3^0^ 
3.08 
3.09 
3.16 


7 

38 

36 


* 7 
17 

21 


20 


- -20- - 


5.10 
5.11 

•5;12 


23 
2? 
35 


9 
39 
38 


t . * ' f 




7.11 
7.14 


-lU 

. 3 


6 
1 




22 

16 . ■ . ' 


8. .04 
8.08 


17 
39 


8 
28 


25 
37 


34 
2H 


9.02 
9.04 
9.06 


29 
5 
28 


36 
11 
31 


22 


23 


10.02 
10.07 
10 . 10 


6 
31 


10. 

18 
32 


'2 

18 
32 


12 
0 

35 - ■ 


11.06 
11.08 


16 
ij 


29 

.4 


13 

■ 8 


19 

- >i3. ;■ 


12.03 
12.04 


30 


25 
5 






20.01 


. 


26 


33 


30 - • 


27.02 
27.03 
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•j 21 
1 

. 26>, 


37 
2 
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1 




1 
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Table lV-9 / 

Results of Ranking Analysis for 
Control Children 



1 - Highest Site Mean P « Fall 

43 « Lowdftt Site Mean S » Spring 



Site 


Ml 


PPV - 




WRTR. 


WRTN 


WM 






F 


S 


F 


S 


.Y p 


S 


F 


S 


F 


S 


F 


S 


2801 


37 


42 


39. 


36 


24 


37 


42 


43 


40 


40 


43 


43 


2802 


28 


38 


27 


30 


31 


41 


37 


42 


7 


30 


25.5 


38 


2803 


24 


29 


11 


15 


17 


24 


27 


25 


5 


31 


25.5 


33 
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Results of Ranking Analysis by Test «> 

In this section we present a brief summary of the 
results of the ranking analysis for each test# An overall 
summary of the ranking analysis with more interpretation 
of the results will be provid^ in the following section. 

In order to estimate the relative effects. of various 
models on a given test, we calculated the fall and spring 
ranks for all Head Start sites. These aife displayed in 
Tables IV-1 through IV-8* Recall that t;he site With the 
highest mean is given rank 1 and the lowest rank 40.^ 
Thus a decrease in rank is evidence that a site has improved 
its position relative to the other He^d Start sites. 

As a rough measure of how much change, has occurred 
overall, we have computed for each test the Spearman rank 
correlation {r\ between the fall and spring rankings. 

s '■ 

The rank correlation measures the degree of similarity 
between two rankings of the same set of objects. Thus 
a value of near 1 would indicate that the relative 
position of the sites ha# changed little from fall to 
spring, implying that model effects are quite homogeneous. 

While we do. not wish the Control results to influence 

V 

our inter-model comparisons/ we are interested in how the 

Control children perf.orm relative to the Head Gtatt children. 

J . ■ - 

He therefore calculated the ranks out of 43 total sites that 
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would have been occupied by the 3 Control sites had they 
been included with the 40 Head Start sites in the analysis. 
These results appear in Table 

Since there are 40 Head Start sites, the expected 
rank fox a randomly selected sit 3 would be 20*5. If the 
NPV sites as a group do not differ from the PV sites, we 
would expect their average rank to be around 20c 5 in both 
fall and spring. By looking at the average ranks for the 
NPV sitesf we can get an idea whether they differ from the 
PV sites in initial level or effectiveness. 

Preschool Inventory . We estimate the reliability of site 
means for the Pai to be between .98 and .99. The fall- 
spring rank correlation is ^90. The mean rank for the 
NPV sites is 19.8 in the fall and 20.5 in the spring. 

Looking now at the individual models, we find. that 
Oregon and Pittsburgh show rank decreases (i.e., improvement) 
in both of their sites, and Par West In all 3 of its sites. 
Thus there is evidence that these 3 models are particularly 
effective in improving PSI scores. No model seems con- 
sistently ineffective, though Bank Street is something of 
^ puzzle. In the fall, one Bank Street site (Tuskegee) 
has the highest mean of all 40 sites, while the other sites 
(Wilmington and Elmira) have the lowest means. The rank 
for Tuskegee slips from 1 to 14 in the spring, while 

o 
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Wilmington and Elmira remain near the bottom. Although the 
^ank Street performance is rather poor, the fact that 
two sites start out so low leads us to suspect that some 
peculiarities of these sites may be more reiaponsible 
than' the model. 

All 3 Control sites experience substantial increases 

in rank from fall to spring. Thi^'implies that Head Start 

■ ' ■ = / ■ ' 

programs were generally more effective than the Control 

"program" in raising PSI scopes. 

■ ' ' / 

Peabody Picture Vocabulary Test * We estimate the site mean 
reliability of the PPV/^o be between .97 and .99. The 
rank correlation betyeen fall and spring is .88. Mean 
ranks for the NPV pitea are 23.3 in the fall and 22.2 in 
the spring. No m9del stands out as particularly effective 
or ineffective in raising PPV scores. Of the Control* 
sites, one deqreafies in rank slightly, and the other two 



Increase slightly 
no less effective 

WRAt Copyin^^^ark^ 



suffers from floor 
model underlying 
inaippropriate. 
subtests as well. 



On the whole, the^ Control sites appear 
than the Head Start sites. 



As explained previously, the WRTC 
effects, so that the classical measurement 
^ur reliability estimates is probably 
s comment applies to the other WRAT 
Thus, although we calculate the site 



Thi 



-■ »' - ■ 

' ■ - '• " 

i . ' ■ ■ 

mean; reliability as .98, we are not sure that this figure 

is really meaningful. The fall-spring rank correlation is 

.85. The mean ranks f or the NPV sites are 26.1 in the fall 

. . " .. . . , , ^ # 

and 22.0 in the spring. The Kansas model stands out strongly, J * 
as both sites improve their positions dramatically. There 
is also a suggestion that the Pittsburgh model may be 
particularly effective^ and the Florida model ineffective. 
All 3 control sites have much worse positions in the spring 
than in the fall* 

WRATc Recognizing Letters ^ Our best estimate of site mean 
reliability is around .96. The rank correlation between 
jfall and spring is only. 60. The mean rank for N'^V sites 
ia 23.8 in both fall and spring. The Arisibha, Kansas^ and 
Pittsburgh i^pdels seem particularly off ective'^^^nd the ^.^^ 
Enabler model particularly ineffective. There i$\a suggestion 
that High/Scope and. Oregon are Ineffective. Of the 3 
Control sites/ one decreases slightly and two increase 
slightly in rank from fall to springs 

WRAy Naming Letters . Site mean reliability is estimated at 
.96. The fall-spring rank correlation is .79* The mean 
ranks for NPV sites are 23.6 in the fall and 24.2 in the . 
spring. Far West and Bank Street seem relatively ineffec- 
tive. The Control sites do very poorly. 
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WHAT Reading Numbers . The estimated site me^m reliability 
is only. 92. The fall-spring rank correlation is .75. 
Mean ranks for the NPV sites are 23.2 in the fall and 22.7 
in the spring. The Oregon, Kansas, and Pittsburgh models 
perform very well. Par West and Florida do poorly. The 
Control sites also perform poorly. 

ITPA Verbal Expyeaslon . We estimate the site mean 
reliability to be between .95 and .98. The fall-spring 
rank correlation Is only .43, indicating either that our 
reliability estimate is Inflated, or that there is con- 
siderable variation in progreim effectiveness iunong sites 
(and possibly models). Mean ranks fpr the NPV sites are 
21.3 in the fall and 18.3 in the spring. The Pittsburgh 
model appears most effective. There is a suggestion that 
Oregon and EDC are also effective, and that the Enabler 
model is ineffective. The ITPA was not administered to 
the Control children. 

ETS . The estimated site mean reliability is .98.' The 
fall-spring rank correlation is .66. Mean ranks for the 
NPV sites are 19.2 in the fall and 10.9 in the spring. 
Oregon, Kansas, and Pittsburgh seem particularly effective, 
High/Scope seems ineffective. The ETS was not administered 
to the Control children. 
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Sviwmary of Ranking Analysis Results 

In Chapter I we stated the three major questions on 

which our analyses will focus. In this section we present 
w 

what evidence the ranking analysis provides bearing on 
these questions* 

/I. To what extent does a Head Start experience 

accelerate the rate at which disadvantaged pre- 
schoolers acquire cognitive skills? 

Our evidence here comes from the performance of the 
Control sites relfitive^ td the Head Start si^^ as a whole 
(PV and NPV) . Of the six tests for which we hay© data on 
both groups, the Control bhildren clearly lose ground 
relative to the Head Start children on four (PSI, WRTC, 
WRTN, WRTD) . On the PPV, Hdad Start and Control children 
perform comparably. The PPV measures very general skills 
which are perhaps not easily taught in a pre-school program, 
On the WRTR, two Control sites (Huntsyille and Sacramento) 
drop from near the bottom to the bottom two rungs. The 
third site (San Jose) has a rank of 27 in the fall and 25 
in the spring. Tables 11-15 and 11-16 indicate, however / 
thcjt because of the ce iling effect/ there is little 
variability in spring site means. Many childreh are at the 
maximum score of 10. Thus the exact ranks have little 
meaning, since a difference of a few tenths of a point may 
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correspond to a large difference in rank. This also ppol^ably 
explains why the rank correlation for the WRTR is only •60, 
The upshot of all this is that we do not have very good 
evidence on whether Head Start is effective in teaching 
the ability to recognize letters, 

2. Are the Planned Variation models, simply by virtue 
of sponsorship, more effective than ordinary non- 
sponsored Head Start progreuns? 

The evidence here is that overall PV and NPV| sites are 
very comparable. The NPV sites as a group perform just 
al>out the way we might expect a randomly select'ed subset 
of 12 out of the 40 Head Start sites to do. In fact, 
according to the theory behind the Wilcoxon test (see e.g., 
Snedecor and Cochran, 1972) , under this null hypothesis 
the mean NPV rank for a given test Wbuld have an approxi- 
mately normal distribution with mean 20.5 and standard 
deviation 2.8. Since fall ^nd spring rankings are npt 
independent wc cannot formally test the significance of 
mean rank changes. It is worth noting, however, that 
the NPV means for all tests, both fall and spring, lie 
comfortably within 2 standard deviations of 20.5. 

3. Are some PV models particularly effective at 

I 

Imparting certain skills? 
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Table IV-10 presents a sumraary baa^d on the inform^'- 
tion in Tables rv-l through IV-9» Therie are a fair number 
of apparently strong positive or negative model "effects.'* 
In terws of tests^ most of these effects (18 out of 22) 
occur in 4 of the 8 tests (PSi, WRTR/wrtD/ ETS) • These 
tests may be more sensitive to program differences. 
In terms of models, it is interesting that of the 15 
positive effectsr 12 are for the •^academic" models (Oregon, 
Kansas, Pittsburgh) , which also show no ne.gative effects. 
Moreover, all 8 ++'s aife for these models. Thus, we have ^ 
at least tentative evildence suggesting that the academic 
models may be generally more effective in transmitting 
academic skills. ThiJ may be the result of a test battery 
more; sensitive to model differences, but w!e withhold final, 
judgment until we have the results of our other analyses. 
Besides the three academic models, the overall performance 
of the Arizona model is also somewhat encouraging* 
Arizona does not do really poorly on any test, and does 
well on the WRTR and ETS. No model does consistently 
poorly across the board » 
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1 Table lv-10 

' SUMMARY OP RELATIVE MODEL EFFECTIVENESS 

/ BASED ON- RANKING AN;a.YSIS* 

. >_ ^ — . . 



f+ Indicates model appears to be highly effective. 
+ Indicates evidence for above average effectiveness. 
Indicates evidence for bo low, average effectiveness. 
— Indicates model appears to be highiy ineffective. 



Model 


PS I 


PPV 


WRTC 




WRTW 


WRTD 


ITPA 


ETS 


Par West 


+ 




i 




















+ 
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/ank Street 


















Oregon 


+ 










++ 




+ 


Kansas 






++ 


+ 4- 




++ 




+ 


High/Scope 




/ 














Plprida 




/ 














EDC 


















Pittsburgh 


+ 






++ 




++ 




+ 


Enable rc 



















* REC not included becausq^ v;ith only one site we felt it 
unfair to drav; any conclusions. 
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Chapter V 

i \ REBIDUAL KNALYSIS 

Introduction 

Thd ranking analysis makes ra^ther niinimal assumptions 
oh the nature of children* s growth processes and how they 
are affected by the HSPV programs. It results in useful; 
inferences about relative model effectiveness/ but does 
not provide a precise numerical itieasure of program effec- 
tiveness* The "residual analysis" described in this chapter 
makes more stringent assumptions on the nature of growth 
processes and tries to provide a precise measure of the 
absolute effect oi a Head start program. 

The basic idea is to estimlite for each child the spring 
test score he would have obtained had he not been in a pre-* 
school program. Comparing this projected spring score with 
his actual fall and spring scores , we can estimate how much 
of his growth is the result of "natifral" maturation and 
how much is a residual effect attributable to the pr^ram 
In which the child was enrolled. 

Smith (1973) used this approach to estimate overall 
Head Start effects. We have refined his method and provided 
more theoretical underpinning. The basic theory of th^ 
residual analysis is presented heuristically in the follow- 
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Ing section « The statistically sophisticated reader Is 

referred to the more mathematical discussion In Appendix D. 

■ ■ ^ ^ ■ - . 

Theory of Regtldual Analysis 

The major assumption underlying the residual analysis 
is that variation displayed by fall test scorej in our 
sample reflects developmental trends over time. More spe-- 
clflcally, suppose we could look at a sub-sample with 
identical values of all background characteristics 
except age. Suppose we observe the mean 

score for such individuals in our sample as a function of 
age. Then we are assuming that the resulting curve is 
very similar to the natural developmental growth curve 
for such children as they grow older. 

^ In general this will be true Unless our san^le has 
selection biases which imply a relationship between age 
and ability. Suppose, for example, that the younger children 
in our sample tend to be particularly clever aa a result 
of the way the sample was selected. Then children who are, 
say, 52 months old at fall testing will on the average do 
better at spring testing (say 6 months later) than those . 
children in our sample who are 58 months old in the fall 

do in the fall. 

\ ' * ' 
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Ab ah introduction tb the "full-blown residual analysis 
we first present a raor6 Intuitive graphical analysis which 
will provide a rough Idea of the^^overall effect ^df Head • 
Start • Suppose we graph the mean outcome 8c6re for a 
particular group of Head Start children as a function of 
age for both fall ^nd spring-* Figure V^-l^ for .example/ 
presents the results on the PSI for all childrerf with no 
prior pre'^-school experience* We have divided the sample 
iDfto 3^onth age groupings and plotted the mean for each 
groups ^ ^^ 



If Head Start had no effect^for these children, we 
would expect the fall and spring curves to be s^milar^ Of 
course, some age groups may turn out to be a bit cleverer 
than others^. and there will be sampling fluctuations so 

that the curves will not be identical # even if Hcdd Start 

*' . ■ /* 

! ' ^ ' 

hoMoXiO effect. Suppose, however, we find that the spring 
curve is consistently above the fall curve. Then unless 
there is a selection Effect in the sample implying a 
consistent negative relationship between ago and ability, 
we have evidence that Head Start' has raised the level of 
the growth curve. The difference between th^ curves pro-r' 
Vides a rough estimate of the value added by Head Start 
over and above that expected on the basis of "natural 
maturation. ^ ^ 
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PSI - FOR ALL CHILDRE N W ITH N O 
"PRIOR PRE-SCHOOL EXPERIEKCI] 



0 



Fftll 



, Spring 



Mean 
Score 

30 



2^ 



20 



15 



10 




pre 4 
post 



46 52 '58 -64 70 76 

16 63 207 .374 3D7 360 257 162 165 .119 52 1 2 1 
3 5 6 19 94 2C4 311 313 300 186 146 124 98 
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For example, according to Figure children with no 

prior preschool between 51 and 53 months of age in the 
fall averaged approximately 11^3 on the PSI, Suppose for 
simplicity that there were 7 months between fall and spring 
testing for all children.* Then we would expect these 
children to obtain an average score in the spring of about 
14,5 without Head Start. Their average spring score was 
in^fact about 16.5. The difference of 2 points represents 
^ yesldual effect, possibly attributable to Head Start, over 
and above the expected natural growth. 

The results of the various graphical analyses we 
carried out are diffiquit to summarize verbally. The 
interested reader is referred to Appendix C, where the 
resulting graphs are presented. 

If in fact certain age groups are cleverer than others, 
such differences in ability may be at least partially 
associated with varioun background characteristics. Thus, 
selection biases can be reduced by carrying out thia 
graphical analysis on various sub-^classes of the total 
sample. For example, one sub-class might be Black males 
with prior pre-sfchool experience. The difficulty, of course, 
is that the more refined we make our subclassif ications 
the smaller the sample sizes become and the more complex 



*The actual time varied from 6 to 9 months. 
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the interpretation • Thus^ the growth curves would be 
estimated very imprecisely • One way to avoid this 
dilenma is to define a mathematical model to describe 
the developmental process v:hich^ if correct^ makes more 
efficient use of the data. This brings us to the full 
residual analysis. 

Our first task is to build a mathematical model to 
enable us to predict the expected value of the test score 
of a child not in any pre -school program on^ the basis of 
his age and other background characteristics. We do this 
by the technique of regression analysis. The details are 
described in the next section. 

Suppose r nowr that we have such a model Then for any 
child in our sample we can compute 6^ r the increase in 
score he would be expected to achi^^ve betv/een fall and 
spring testing on the basis of natural maturation. This 
is doJfie by up-dating his age the appropriate number of 
months r Reaving other background variables unchanged/ 
and calculating the effect this would have according to our 
model . 

For example, let Y and represent fall and spring 
test scores respectively. Let AGE 1 be the age at fall 
testing^ and HOMED be the number of v^^i^^ of mother^ s 
education. Suppose our regression model is given by 



ERLC 



123. 



Y» « 4 + .2 X AGE + .1 K HOMED + e (5.1) 

where e represents random error. In this case A Is simply 
.2 times the nxnnber of months between fall and pprlng testing. 

Having calculated ^ , we c^in caroulate an estimate of 
the child's expected spring scol^e in two reasonable ways. 
One is to simply add 6 to the observed fall score (Method 1) . 
For example, suppose we have the model described by equa- 
tion (5.1)/ that there are 7 months between tests, and that 
for a given child AGEl is 50, HOMED is 10, and his fall 
score is 17. Then: 

Method 1 Expected Spring Score = Y + A =^ 17 + .2 X 7 18.4 

The second way (Method 2) to estimate the expected 
score usee <:he regression model directly, we slnply 
substitute the appropriate values of background variables 
(including the age at spring testing) in our regression 
equation. We obtain an estimate of the expected score which we 
shall call ^" . Note that if we do the same thing using the 
age at fall testing, we obtain a predicte d fall score 1^, 
which is what we would expect the individual to have achieved 
in the fall on the basis of his background characteristics. 
Because the aregression model is linear, J:he Method 2 
expected spring score is mathematically equivalent to adding 
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to the predicted f^ll score rather W>an the actual 
fall score. 

In our example 

? « 4 + .2 X St) + .1 X 10 « 15 
f'- 4 + .2 X 57 + .1 X 10 16,4 



Thus 



Method 2 Expected Spring Score" 
« 4 + .2k 57 + .1 X 10 
« (4 + .2| X 50 + .1 + 10) + ,2X7 



Finally/ the residual attributable to the child's 
program for each method is the difference bjetween the 
observed and expectjed spring scores. Thus, for Method 1 
we have 



rj^> Y' - (Y + ^ ) 



and for Method 2 



Y' - (Y+A ) 



Thus in bur ex£unple, suppose the spring score is 19. Then 
we have 
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- 19 - (17 + 1.4) + .6 

«« 1? - 16.4 = 2.6 

. Note that if the regression model were exact (perfect 
. prediction, no error variance) and the test perfectly 
reliable, then the two methods would be equivalent and 
would perfectly estimate the "true" effect. In a real 
situation the tradeoff between the two methods hinges on 
whether the observed or predicted fall test is a better 
estimate of the true fall test score. Roughly speakihg, 
higher test reliability favors Method 1 (using the observed 
fall score), while more accurate regression equations favor 
Method 2 (using the predicted fall score). The existence 
of this tradeoff suggests that a weighted combination of 
the estimates provided by the two methods may be optimal. 
We shall have more to say about the appropriate weighting 
in the next section. A more theoretical discussion of 
the issue is presented in Appendix D. 

We can compare various Head Start programs by estimating 
the mean residuals for their program groups and comparing 
them. Since the residuals reflect the increase in score 
beyond that expected on the basis of natural maturation, 
they provide an absolute measure of program effect. Thus 
the "effect" for the Control "program" can also be calculated 
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and used as a check on our methods. If the Control children 
really are similar to Head Start children in the fall, 
we would expect thdr residuals to average close to zero* 
That is I we would exppct their spring scores to reflect 
only maturation, if the Control residuals are substantial, 
there are at least three possible explanations. There may 
be some sort of test sensitization or pract^ice effect; 
some of the Control children might actually be involved 
in a preschool program? or there may be a selection bias 
in the fall sample causing us to underestimate the slope 
of the growth curve. If the slope is underestimated, we 
underestimate the projected spring scores and overestimate 
the residuals. The upshot of this discussion is that small 
residuals for the Controls is evidence that our technique 
is working as it should. Lar^e residuals are troubling, 
since they may mean either that the analysis is in some way 
incorrect or that our Control children fail in some way to 
be legitimate controls« 

In concluding this section, let us summarize the two 
major assumptions on which the residual analysis is based. 

Assumption 1 ; The relationship between fall test 
score and age in our sample accurately reflects the develop- 
mental process occurring over time. 

Assumption 2 t A linear regression model adequately 
represents the relationship between fall test score and 
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background characteristics (including age) ^ 

c 

In the following section we describe the way in which 
the regression models used in the residual analysis were 
developed • 

. {^egression Models 

In this section wja describe the d^ivation'of the 
regression equations necessary to implement the analysis 
approach described in the previous section, Thes^ equations 
are also of some interest in their own right as descriptive 
statements about the developmental process for young children • 

We attempted to build the best /possible model to describe 
the relationship between expected c^Utcome scores and our 

measured background character isticjB in the absence of 

! ' ' 

program effects. To do this, we aried to explain as much 

variance as possible in the pre*?^est scores with the measured 

background characteristics. By^ /performing regression 

analyses using fall scores for the entire Head Start 

sample as the dependent variable, we eliminatei program 

/ ' 

effects. Moreover, the sampl^ size was large enough to ensure 
accurate estimation. /' 

We began with some exploratory regression analysis 
involving as independent variables a wide variety of back- 

■ 

/ 
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ground variables, Including age (AGE 1) , sex (SEX) , 
, mother's education (MOMED) , ethnicity (ETHBL, ETHWHITE),' 
family income (PAMINC) r first language (FLANG) , prior 
preschool experience (PS, PSMNTHS) , household size 
(HHSI2E), and sex of the head of household (SEXHH) • From > 
here on we will often for convenience/ refer to the abbre- 
viations for variables defined in Appendix A, where the 
exact coding for all variables can also be found. 

For all tests it was found that restrict;ing attention 

to AGEl, SEX, HOMED, FLANG, and the ethnicity and prior > 

# > 

2 

preschool variables lost very little in terms of R , the 
proportion of fall score variance which could be explained. 
We were concerned that the effects of first language and 
pribr preschool might be particularly ccwnplex. We therefore 
decided to divide the total sample into the following 4 
exclusive groups, and to fit a separate regression model for 
each group; 

Group Is PLANG : First language not English 

Groyp 2s PSNHS : Non -Head Start prior preschool experience 

Groiip 3: NOPS ! No prior preschool experience 

Group 4: PSHS : Prior Head Start experience \ 

There are a few children with prior preschool ex- 
perience whose first language was not English, but since ^ 



ERIC 



129, 



the sample was rather small to begin with, we did not 
separate them out* A dummy variable indicating previous 
pr^chool experience was, however, included in the 
regression for Group 1* 

In all 4 groups we eliminated from the analysis the 
very few children who were not Black, White, or Spanish 
American (Mexican American or Puerto Rica^n) . Since Group 1 
was comprised almost entirely of Spanish Americans, we felt 
the results would be more meaningful if the others were 
eliminated* The analyses for Groups 2 and 3 contain dummy 
variables for both Black (ETHBL) and White (ETHWHITE) . 
Group 4 had nearly all Blacks and Whites, and the 'analysis 
included a dummy variable for Black only. 

Note that for children with some prior pieschool 
experience, their "natural" developmental process may 
have been altered in a variety of ways* Thus, although our 
approach is probably most suitable for Group 3 (which in- 
cidentally comprises about 2^3 of the sample) , we have 
decided to carry through the analysis for the other groups 
as well, but to be somewhat careful in interpreting the 
results. 

In addition to the basic variables themselves, all 
2-way interactions among them were also considered. Several 

i analyses were run for each of the 4 groups in an attempt to 

\ 
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obtain a large R with as few variables as possible. The 

final set of equations selected are presented in' Tables 

V-1 through V--8. Each row of these tables represents one 

, ' ^ . •» 

equation. Variables are specified by the colujrm headings • 

The entries in any column are the regression coefficients 

associated with the specified variable. ^For the PSI for 

children with no prior pre-school experience, for example, 

the regression equation reads: 

Expected PSI score = 6.518 + .0544 X AGE + .8883 X SEX + 
1.549 X HOMED + .0341 X AGE 1 X MOMED - 1.977 X ETHWHITE 
- .1931 X ETHBL X HOMED + .2596 X ETHWHITE X MOMED. 

Before discussing these equations in more detail, we 
presfent two digressions which may aid the reader in inter- 
preting them. First we consider the interpretation of the^ 

coefficient of an interaction variable, and then the sig- 
2 

nificance of R , the proportion of variance explained by 
our independent variables* 



I nterpretation of Interaction Coefficients 

Consider any two independent variables, such as AGEl 
and SEX. The interaction variable is simply the product 
of tjie two, e.g. AGEl X SEX. The coefficient for such a 

i 
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variable may be Interpreted as the effect that a unit 
increase in either of the variables has cn the effect 

s 

of the other variable. For example, the coefficient of 
AGEl X SEX* in the PSI Group 1 equation is .3764. ThuB 
we can say that, all else being equal, the rate of increase 
of PSI with age is ,3764 points per month higher for girls 
than for boys. can equivalently say that the advantage 
of being a girl rather than a boy increases by .3764 
points per additional month of age (ot more accurately here 
the disadvantage descreases) . 

If an interaction coefficient is statistically sig- 
nificant, it means that the combination of the two variables 
has an effect over and above what can be adequately described 
by simple additive effects. In our example, the adv^antage 
which accrues to an older girl is greater than the sum 
of the effects of being a girl and being older. 

Significance of Explained Variance (R ) 

For the residual ^analysis to be valid, it is not 

2 

necessary that R be very large. Roughly sjpeaklng, the 
2 

larger R the smaller will be the variance of the resulting 



-^SEX is coded 1 for girls and 0 for boys. 



ERLC 



140. 



residuals, however, and the more precise the analysis will 

be. In many contexts in which regression analysis is 
2 

us;ed, R setves as a measure of the strength of the 
independent variables as predictors of the dependent var- 
iable. In the present context, the value of is deter- 
mined by three factors, in addition, of course, to random 
fluctuations. 

First there is the importance, or strength, of effects 
on the dependent variable attributable to the independent 
variables, as measured by the regression coefficients. 
Second, there is the variability in our sample. To take 
an extreme example, if there were ^no Blacks in our sample, 
the ETHBL dummy variable could explain no outcome variance 
regardless of the true effect of being Black. 

If the distribution of the independent variables is 
similar to that in the population to which we wish to 
generalize our regression results, the R^ for that popu- 
lation will be similar. If the distribution is different, 
2 

R may 'be quite diffefrent, even if the effects are the 
saritb^ The values of the regression coefficients do not 
depend on the distribution of the independent variables in 
our sample, although our 'ability to* estimate these coeffi- 
cients accurately might . 

2 

The third factor on which R depends is the reliability of 
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the outcome measure* . A test score Y can be thought of as 
containing a t^-ue component T corresponding to a stable 
character is ti6 of interest, end a random error component e. 
Part of the variance of the true part T can be related to 



measured ir)dependent variables via regression analysis. j 

The highe^\he reliability of Y, the larger the proportion / 

^ • /■■' / - ■ / 

of its variance attributable to T; hence the more potentially / 
/ ^ " ' / 

explain?(ble variance* Thus, the reliability sets an upper ^ 

bound pn the proportion of variance explainable. 

The upshot of this discussion is that although 

/ ■ ' 2 

meocd^nizing R is desirable for maximum precision, the value 

/ 2 

ot ^ is determined by a complex interaction of factors, 
making its interpretation difficult. The interpretation 
6f the coefficients determining our regressid^ model, on 
the other hand, is straightforward, and for large sample 
sizes the estimation of these coefficients should be accurate. 

Let us look now at the regression models. It is 
difficult bo summarize all the implications of these 
equations in any simple way. To measure the net effect of 
a variable in a particular equation, it is necessary to consider 
also all interaction variables with non-zero coefficients 
which involve this variable. Suppose, for example, we are 



* We assume here' that the independent variable- have perfect 
reliability* Unreliability in the independent variables 
introduces further complications.. See Chapt, VI. 
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Interested in the effect of age on PSI score for Group 1 
children* Then a one month increase in age corresponds 
to an average PSI increase of ; 

A «:,3764 X SEX + .0363 X MOMED 

^ - . „ 

Thus the effect of age in this case depends on the values 

of SEX and MOMED. For a boy with MOMED « 12, 

^ ^ •3764 X 0 + .0363 x 12 = .436. 

For a girl with MOMED « 10 

A ^ .3764 X 1 + ,0363 X 10 = .739. 

At first sight it may appear that the equations for 
the four groups on the same test differ wildly. Closer 
inspection of the net effects of each of the variables re-^ 
veals that over the range of values found our sample 
the equations are quite similar, 

With 8 tests and 4 groups^ we might not expect to find 
consistent patterns in the effect of a given /variable across 
the various equations. Surprisingly^ certain patterns do 
emerge. These are summarized in Table V-9 . 

It seems that SEX and interactions involving it tend 
not to show consistent patterns, while the effects of age, 
ethnicity, and mother's education are quite consistent. 

In concluding this section, let us state that we believe 
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Effects of Background Varlablea on Fall Scores 



Variable 



Net Effect , 



ICE 1 

SEX 

HOMED 

ASBl X SEX 
PGEl X vom) 
SEX X vcm> 

ASEl X BTHBL 
EfTOBL X MGMED 
ETHWHITE X MOMED 
ETHBL X SEX 
ETHWHUE X SEX 



Nearly alweys + 

No consistent pattern except in grp* 1 where 

Generally + except sometimes for youi^g 
blade dUldren and girls in grp. I 

Nearly always + \ 
Nearly always + 

Generally + except for Group 1 
Generally - 
Generally - \ 
Gerverally + 

No consistent pattern 
No consistent pattern 



144. 



the ycgresslon models described In Tables V-1 though V-8 
to be reasonably accurate# concise mathematical descriptions 
of the relationship between test scores and various background 
characteristics in the absence of pre-school intervention for 
Head Start-age children * In carrying out the residual 
analyses in the following section, we shall use these equations 
in this way* We recognize that to be formally correct, ^ 
we should in some way take into account sampling errors in 
the estimation of the regression coefficients* This would, 
however, make the analysis almost impossibly complex and 
add little to our confidence in the results, 

Jm plementation of Residual Analysis 

In this section w% present brief descriptions of the 
various ways in which the residual analysis was implemented 
and the basic tables of results. At the outset, let us 
remark that the residual analysis was carried out for only 
6 of the 8 tests. Since the WRAT Recognizing Letters and 

0 

Naming Letters subtests generally had such low values of R , 
we felt the regression models were of questionable validity, 
particularly in light of the floor and ceiling effects* l*ha 
WRAT Reading Numbers seemed of borderline acceptability, ^ 
but we decided to include it. 
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For the six tests / we calculated for each child in 
the sample with the necessary data a Method 1 and Method 
2 residual* As explained above, to compute the Method 1 
residual it was first necessary to calculate A # the 

i * ' 

expected, increment. This was found by incrementing the | 

. i 

child's age by the actual nuxnber of months between. his ^all 
and spring tests and calculating the effect this would I 
have according to the appropriate equation. Adding^ td 
the fall test score, we obtained the predicted spring 
score in the absence of a pre-*school progr2Lm. Finally, 
the residual was found by subtracting the {predicted scor:e 
from the actual spring score. To find Method 2 residuals 
we obtained the predicted spring score by simply substi- 
tuting the child *s age at spring test time in the appro- 
priate equation. 

In Tables V-10 through V-15, we present the results 

% 

of the Method 1 analysis by model. For each model, we 
present the mean fall score the mean spring scor^ , 

the mean expected increment Z , the mean residual ^ , 

[1 

and the samplie size. Thus, we partition the mean g^in 
7' - ? into two parts; the increase we would expect on 
the basis of natural maturation, and the residuaJL increase 
over and above this which may be attributable to the effect 
of the model. 
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Method 2 results are prefifented In Tables V-16 t\irough 
V-21.' Here we present the mean predicted fall score 5^ 
based on the regression equations, the mean spring scores « 
the mean expected increment^ mean residual r^ and sample 
size. Spring score and expected increment means would be 
ideotical for the two methods axcept for the fact that the 
ahlilYses . are based on slightly different samples. .Alsor 
because of the nature of least-^squares regression analysis / 
the observed and predicted fall score means for the entire 
sample would be identical » For particular programs, 
however^ they might differ, since a program group may on 
the average do better or worse in the fall than we would 
expect on the basis of background characteristics. 

We will find an apparent model effect more credible 
if it is consistent across all the sites in t^e model. To 
check this, we have computed the mean residual by both 
methods for each site. These results appear in Tables 
V-22 through V-27. 

Having perfozrmed these analyses it oooiirred to us 
^that ti^ere might be some optimal way of combining the two 
methods to obtain a better estimate of the mean residual 
for each model. It seemed logical to consider weighted 
averages of the Method 1 and Method 2 residuals. If r^ 
represents the residual from Method 1 and r^ from Method 
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2# we can consider all combinations r of the form 
♦ 

r « w rj^ (l-w) rj i / 0< w t 1; 

For any value of w we can compute the combined residuals 

and use these as outcome measures in a one-^way analysis 

of variance with programs as factors. It seems reasonable 

* 

to try different I'^iuos of w and select that value w . 
which minimizes t^e within^'group mean square. A more 
detailed rationale is presented in Appendix D."" The re- 
suiting ANOVA also provides a measure of . the statistical 

significance of model differences'. The results of the 

\ 

"Combined" residual analysis are presented in Table V-vJ8. 
We were particularly interested in comparisons among the 
means for the PV children as a whole, the NPV children, 
and the Control children. The results of various t-tests 
based on the ANOVA are presented in Table y-29. 

We were somewhat concerned by the fact that the theory 
underlying the residual analysis might be less appropriate 
for children with some prior preschool experience. Since 
the models vary somewhat in their proportions of children 
in the four groups for whi6h separate regressions were run, 
biased conipari^tons may result. As a check, we computed 
model means and one-way ANOVA' s for the* four groups separately. 
Thesb results are presented in Tables V-30 through V-^35. 
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RESULTS OF 


RESIDUAL ANALYSIS 










J&TIIOD 1 
felt. / 


i 


' 




Program 


Fall /' 
7 


Spring 
,Y' . 


Expected 
Increment 


^ , Residual 


Sample 
Size , 

n ' 


Far wost 


14 • o9 


20.79 


3.22 


2 . o7 


126 


Arizona 


15 . 95 


20.56 


3.24 


1 . 37 


1 Oil 
184 


BanK bcreec 


14.11 


16.91 


3.08 


- • 28 


225 


Oregon 


.17.06 


23.26 


3.36 


2*84 


141 


Kansas 
A 


14 • lU 


18.86 


2.77 


1 . 9^ 


9 / 7 


High/Scope 


16 • 05 


19.81 


3.13 


.63 


169 


Fiorlaa 


13.96 


. 18.52 


2.82 


1 1 A 

1 . 74 


13o 




1 3 OQ 
X4 . OO 


19.55 


2.68 






Pittsburgh 


13 . 37 


, 19.55 


3.40 


2 . 77 


A A 

99 • 


REC 


12.45 


17.76 


2.78 


2.53 . 


67 


Enablers 


14.59 


19.62 


/ 3.09 


i.94 


181 


Control 


12.22 


15.12 


2.51 


.39 


105 


tlPV 


14 • 92 


19.23 


•3.01 


1 • 31 


^ A A 

609,*5^ 


TOTAL PV 


14 . 83 


19.54 


3.07 


1. 66 


n t* o O 
15 OO 


TOTAL 


14.74 


19.26 


3.03 


1.50 


2302 
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METHOD 1 

-to; — 



Program 


Fall 
7 


bpring 


Expected 
Increment 


Residual 
^1- 


Sample 
Size 
n 


f d ^ f f ^ 0 W 


37. 76 


48.40 


6 .18 


4.46 


156 


Arizona 


35.14 


45,53 


5. 87 


4.52 


190 




27. 73 


37. 46 


3*94 


5 80 


241 




35.47 


46. 06 


4.69 


5.91 


131 


Kansas 


31, }7 


41.69 


4.7fl 


5.74 


94 


Hloh/Scooe 


35,55 


44.14 


5.63 


2.96 


161 


Florida 


30.28 


41.15 


4; 63 


6.24 


141 


EDC 


29.57 


39 . 75 


3.60 


6.58 


160 


Pittsburgh 


32.13. 


45.92 


7.40 


6.39 


109 . 


REC 


29.28 


43.07 


' 4,08 


9.72 


69 


Enablors 


34.80 


43.29 


5.18 


3.32 


195 


Control 


28.50 


39.26 


'4.70 


6.C7 


106 


NPV 

41 IT V 


29 64 


41.50 


4 . 76 


7. 10 


592 


TOTAL PV 


32.70 


43.02 


5.04 


5.27 


1647 


TOTAL 


31.74 


42.47 


4.96 


5.77 


2345 






1 « 






1 
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Program 


Pall 


Spring 

■■5 

y ' /. ; 


Expected 
Increment ■ 

1 


Residual 

^1 


Sample 
Size 
n 


Par West 


1.93 


4.85 


1.39 


1.53 


123 


Arizona 


1.84 


5.33 


1.28 


2.21 


194 


Bank Street 


1.90 


4.28 , 


1.04 


1.34 


243 


Oregon 


3.66 


8.04 


1.24 


3.14 


138 


Kansas 


1.23 

< 


6.47 


.88 


4.36 


101 


High/Scope 


2.08 


5.58 


1.19 


2.31 


178 


Florida 


2.43 


5.23 


1.01. 


1.80 


103 


EDC 1 


2.43 


5.91 


.78 


2.69 


169 


^4^tsburgh 


1.18 


.4.17 


l.!i9 


1,42 


101 


REC 


.81 


3.48 


1.19 


1.49; 


77 


Enable rs 


2.02 


5.21 


1.21 


1.98 


193 


Control 


1.3S 


2.74 


1.09 


.27 


65 


NPV 


1.85 


4.95 


1.11 


1.99 


606 


TOTAL PV 


2.03 


5. 33 


1.15 


2.16 


1620 ' 


TOTAL 


1.96 


5.14 


1.14 


1 

^ 2.05,, 


^ 2311 
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HETHOD 1 
— 



Program 


Fall 


Spring 


Expected 


Residual 


1 

Sample 








^ 1 1 w J- dlKS J I u 








7 


7' 




^1 


n 


Far West 


. 87 


2.09 


. 40 


. 82 


126 


At! yon a 


76 


2. 24 




1 12 


195 


Bank Street 


.65 


1. 41 


. 30 


.47 


243 




78 




26 


2 65 


14*^ 

X *l .J 


V An fi A Q 


» <j 


2 64 


25 


1 84 






64 


1 68 




71 


1 ftl 

X o ^ 


Pi o Y"i Ha 


84 


1 67 


26 


57 


XI/ o 


EDC 


.73 


2.18 


,22 


1.24 


169 


PIn tF:: i;rgh 


.46 


2.07 


.48 


1.14 


101 


REC 


.23 


1.31 


.28 


.80 


77 


Enablers 


.67 


1.74 


.34 


.74 


193 




40 








It "A 

; 

t 


NPV 


.53 


1.72 


.29 




607 


TOTAL PV 

• > 


.67 

* 


2.03 


.32 


1.05 


1634 


TOTAL 


.63 


1.91 


.31 


.98 . 

J 


2326 
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TABLE V-14 

RESULTS OF RESIDUAL ANALYSIS 

METHOD 1 
I»A 



Program 


Pall 
Y 


Spring 
Y' 


E3q>ected-- 
Inci:eni6nt: 
A 


Residual 
^1 


Sample 
Size 
n 


Par West 


11 . 83 


15 . 49 


2 .40 


1 26 


72 


Arizona 


14.65 


17. 3i 


2.16 


.49 


72 ■ 


Bank Street 


9.28 


13. 31 


1. 38 


2.66 


99 


OrGaon 


12 . 89 


16.96 


1.43 


2 64 


54 


Kansas 


10. 36 


14.19 


1.63 


2. 20 


36 


Hi oh /S cone 


10 97 


14.21 


2 19 


1 04 


73 


Florida 


11.27 


16.04 


1.74 


3.03 


48 


EDC 


12.35 


16.92 


1.52 


3.06 


63 


Pittsburgh 


8.82 


15.57 


2.93 


3.82 


51 


REC 


12.81 


14.44 


1.58 


.046 


32 


Enablers 


12.20 


14.13 ^ 


1.91 


.031 


112 


Control^^ 






















NPV 


10.93 


15. 70 


1.65 


3. 12 


248 


TOTAL PV 


11.57 


15.18 


1.89 


1.71 


712 


TOTAL 


11.41 


15.32 


1.83 


2.08 


960 



ERIC 

I I " ' I I 



TABLE V-15 

153. 

RESULTS OF RESIDUAL ANALYSIS 

^ 

METHOD 1 
-TTS 



Program 


Fall 


Spring 


Expected 


Residual 


Sample 












ox At? 




Y 


Y' 




^1 


n 


Par* Wesii 


8. 83 


13. 26 


2 34 




^8 


Arizona 


8.15 


13.90 


2 13 


3 63 


/ 7 3 


Bank Street 


7.67 


11.67 


1 73 


0 27 


87 




11 72 


16 '^2 


1 7*5 


9 8^ 

^ • 




Kaoisas 


7. 06 


13.29 


1. 84 


4. 39 


35 


H i nh /55 r»rkn#* 


Q 88 


12.21 


2 28 


07 




Florida 


9.16 


12.61 


1.85 


1.59 


43 


EDC ^ ^ 


11.47 


13.88 


1.55 


.85 


84 


Pittsburgh 


8.28 


13.18 


2.58 


2.32 


40 


REC 


10.47 


„ 11.40 


2.03 


-1.10 


30 


En able rs 


11.47 


13.22 


2.25 


-.50 


68 


Control 






















NPV 


9.42 


12.61 


1.82 


1 . 37 


253 


TOTAL PV 


9,46 


13.17 


2.02 


1.68 


594 


TOTAL 


9.45 


13.02 


1.97 


1.60 


847 



ERIC 



TABLE V-16 
RESUliTS OF RESIDUAL ANALYSIS 
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METHOD 2 
"TSI 



Prograin 


Predicted 
Fall 

A 

Y 


Spring 
7\ 


Expected 
Increment 


Residual 


Sample 
Size 
n 


Far West 


14 . 85 


20.59 


3.22 


2.53 


135 


Arizona 


15.14 


20.08 


3.24 


1.69 


197 


Bank Street ^ 


13.76 


16.55 


3.08 


-.29 


242 


Oregon 


16.24 


23.13 


3.30 


3.58 


148 


Kansas 


12. 53 


18. 72 


2.76 


3.43 


99 


High /Scope 


14.76 


19.46 


3. 14 


1.56 


184 


Florida 


14.07 


17.80 


2.83 


.90 


154 


EDC 


14.96 


19.28. 


2. 71 


1.62 


170 


Pittsburgh 


13.83 


19.34 


3.41 


2.11 


102 


REC 


12.59 


16.99 


2.83 


1.57 


78 


Enablers 


14.64 


19.33 


3.10 


1.59 


191 


Control 


12.07 


13.94 


2.51 


-.65 


124 


NPV 


14.52 


19.08 


3.02 


1.45 


636 


TOTAL PV 


14.45 


19.17 


3,07 


1.66 


1700 


TOTAL 


14.38 


18.89 


3.03 


1.49 


2460 



ERLC 

I 



TABLE V-17 

RESULTS OF RESIDUAL ANALYSIS 

METHOD 2 
•-T^ 



XiJ A> 1 1 


Fall 

A 

Y 




Incremont, 


Kosiauai 


Sample 
Size 

n , 


Far West 


3^^.58 


i^8.33 


6.32 


7.1*2 


131 


Arizona 


33.^18 


^5.05 


5.83 


5. 711 


189 


Bank Street 


28.77 


37.32 


1I.05 


1^.50 


2118 


Oregon 


31.37 


^5,98 


H.H6 


10.15 


133 


Kcmsaa 


28.90 


iil.il5 


11.78 


7.77 


98 . 


High/Scope 


33.67 


il3.60 


5.58 


H.35 


178 


riorlda 


30.63 


39.58 


I1.I15 


1| .51 


11j6 


EDC 


30.85 


. 39.^15 


3.61 


11.98 


166 


Pittsburgh 


32.71 


^5.26 


7.55 


5.01 


99 


REC 


28.67 


I12.38 


I1.O6 


9.65 




Enablers 


32.75 


Il3»il9 


5.02 


5.72 


181| 


Control 


27.99 


37.97 


'1.75 


5.23 


117 


NPV 


31.30 


ill. 81 


l\ . 76 ' 


5.75 


604 


TOTAL PV 


31.57 


il2.51 


11.99 


5.95 


16H6 


TOTAL 


31.33 


i|2.11 


n .92 


5.87 


2367 
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TABLE V- I 8 

156. 

RESULTS OF RESIDUAL ANALYSIS 

METHOD 2 
WRM 



Program 


Predicted 
Fall 

Y 


Spring 


Exoected 

Increment 
X 


4 \W i3 X VA Ct 

2 


size 
n 


Far West 


1.92 


4.75 


1.39 


1.44 


130 


Ariiona 


1.91 


5.22 


1.30 


2.01 


205 


Bank Street 


1.69 


4.15 


1.03 


1.43 


254 


Oregon 


2.94 


8.02 


1.21 


3.87 


145 


Kansas 


1.27 


6.34 


.87 


4.20 


10 3 


High /Scope 


2.02 


5.39 


1.18 


2 . 19 


190 


Florida 


1.91 


4.99 


100 






EDC 


2.26 


5.86 


.79 


2.81 


174 


Pittsburgh 


1.03 


4. 13 


1.57 


1 52 


10? 

X \J £t 


REC 


1.35 


3.42 


1.21 


.86 


81 


Enablers 


2.06 


5,10 


1.21 

» 


1.84 


202 


Control 


.88 


2.47 


1.04 


.56 


97 


NPV 


2.02 


4.90 


1.10 


1.79 


660 


TOTAL PV 


1.91 


5.22 


. 1.14 


2.17 


1733 


rOT/iL 


1.90 


5.03 


1.13 


2.01 


2490 
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TABLE V-19 
RESULTS OF RESIDUAL ANALYSIS 



157. 



METHOD 2 



Program 


Predicted 
Fall 

Y 




Increment 
r 


l\6oXQUai 


Saitple 
Size 
n' 


Par West 


.70 


2.07 


.40 


.97 


129 


Arizona 


.68 


2.18 


.36 


1.14 


204 


Bank Street' v 


.54 


1 .39 


.30 


.56 


254 


Oregon 


.69 


3.65 * 


.26 


2.70 


145 


Kansas 


.44 


2.62 


.26 


1.93 


103 


High/ScopQ 


.71 


1.68 


.33 


.64 


1^0 


Florida 


.57 


1.61 


.27 


'77 




EDC 


.63 


2.16 


.22 


1. 31 


/ 174 


Pittsburgh 


. 52 




A Q 


1 f\C. 


102 


REC 


.48 


132 


.29 


.55 


81 


Enablers 


.70 


1.68 


.34 


.64 


202 


Control 


.42 


.74 


.26 


.07 


97 


NPV 


.62 


1.75 


.29 


.84 


660 


TOTAL PV 


0.61 


1.99 


0. 32 


1.05 


1731 


TOTAL 


.61 


1. 88 


.31 


.96 


2488 
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TABLE V-20 
RESULTS OF RESIDUAL ANALYSIS 
METHOD' 2 



158. 



Program 


f . 

Predicted 
Fall 

Y 


Spring 


Expected 
Increment 
I 


Residual 
^2 


Sample 
Size 
n 


Par West / 


11.57 


16.37 


2. 39 


2.42 


59 


Arizona 


11.86 


16.48 


2.22 


2.41 


73 


Bank Street 


10.76 


13.08 


1.40 


.93 


98 


Oregon 


12.58 


17.04 


1.50 


2.96 


55 


Kansas 


10.55 


14. 19 


1.63 


2.02 


36 


High/dcope 


11.56 


13.96 


2.17 


.23 


76 


Florida 


11. 40 


14.72 


1.78 


1.54 




EDC 


1,2 . 0 5 


16.68 


1.51 


3.12 


65 


Pittsburgh 


10.23 


15.55 


3.01 


2.31 


44 


REC 


10.14 


13.47 


1.64 


1.69 


36 


Enablers 


11.68 


14.03 


1.89 


.46 


104 


Control 

























NPV 


11.48 


15.44 


1.65 


2.32 


273 


TOTAL PV 


11.39 


14.95 


1.90 


1.65 


703 


TOTAL 


11.42 


15.09 


1.83 


1.84 


976 
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TABLE V-21 
BESULTS OF RESIDUAL ANALYSIS 



159. 



METHOD 2 

m 



Program 


Predicted 
Fall 

Y 


Spring 
7' 


Increment 
T 




oanipxe 

Size^. 
n 


Far West 


8.97 


12.88 


2.46 


1.46 


134 


Arizona 


9.39 


13.23 


2.28 


1.56 


202 


Bank Street 


8.51 


11.53 


1.81 


1.21 


234 


Oregon 


10.3? 


16.26 


1.77 


4.11 


148 


Kansas 


7.92 


12.98 


1.73 


3.33 


102 


High/Scope 


9.39 


12.12 


2.20 


.53 


184 


Florida 


8.92 


12.14 


1.77 


^ 1.44 


152 " 


EDC 


9.69 


13.57 


1.76 


2.12 


169 


Pittsburgh 


7.86 


11.86 


2.59 


1.41 


102 


REC 


8.06 


10.94 


2.01 


.86 


77 


Enablers 


9.63 


12.61 


2.21 


.78 


186 


Control 















NPV 


9.21 


12.31 


1.89 


1.21 


638 


TOTAL PV 


9.08 


12.77 


2.05 


1.63 


1690 


TpTAL 


9.12 


12.65 


2.00 


1.52 


2328 
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TABLE V-22 160. 

RESULTS OF RESIDUAL ANALYSIS 
BY SITE 



PSI ■ 







V 


NPV 


SITE 


METHOD 1 


METHOD 2 


METHOD 1 


METHOD 2 


02.02 

02.09 
02.13 


2 . 66 
2.91 
2.93 


4.83 
2.00 
1.91 


m 




03.08 
03.09 
03. le 


1 .09 
1 . 59 
1 .49 


2 .20 
.17 
2.44 


.46 

■ ■\ 


1.69 


05 . 01 
05.10 
05.11 
05.12 


-2. 32 
.43 
1.77 


.11 

- .96 

- .07 






0/ . 11 

07.14 
07vl9 


2 . 29 
3.43 


3.71 
3.45 


:7r — 

-2 .02 


1 .80 
1.12 


08. 02 
08.04 
08« 08 


4") 

5.02 
-1.51 


5 . 48 

^ 1 . U o 


2 .75 
1 . 18 


.97 
1 ,33 


09« 02 
09^ 04 
09.06 

o9ao 


, i C 

♦ / b 
1 •SI 


1 .17 


1.81 


2.10 


Id. 02 
10.07 
10.10 


* 1.15; 

70 
3.42 


.15 
1.31 
1,02 


2.35 
.14 

: ..7fi 


2 . 95 
2.61 
—LB '— 


11.06 

X Jl • UO 

11.08 


lie 

. 2 .54 


1 . «? D 

1 .81 


- .04 


.38 


12.03 
12.04 


3.15 
1 .85 


1.16 
4.67 






20.61 




-, . ,1 .S7 . ■. 


4 . 18 


2,p8 












27.61 ' 

27.02 

27.03 

27.04 

27.05 


1 .66 
3.89 
2.95 
.47 


1 . 57 
. 89 

3.09 
.46 






26.61 
28.03 
28.02 


.56 
.97 
.04 


-f.()6 
. 16 
.20 







TABLE V-23 



161. 



RESULTS OF RESIDUAL ANALYSIS 
BY SITE ^, 

PPV 









NPV 


SITE 


METHOD 1 


METHOD 2 


, (METHOD 1 


METHOD 2 


02.02 

02.09 
02.13 


CCA 

1,28 
6.19 


10.36 
6.00 
7.16 




* 


03.08 
;)3.09 
03.16 


J 4. 7 4 
4.59, 
4 .26 


4.13 
4.71 
7.89 


5.83 


6.21 


» U J. 

05.10 
05.11 
05.12 


4.76 
7.87 
4 .90 


3.69 
6.27 
3.72 






U / . 11 

07.14 
07.19 


6, 89 

.5 .00 


7.66 
12,26 


6'. 4 6 
17.28 


7 .49 
7.35 


Uo • 

08.04 

08 • 08 '. 




9.18 

6,,5p, 


5.39 
3.57 


3.84 
1.78 


09*0^ 

A A A il 

uy • U4 
09^06 
09.10 


3.70 
7.18 


1.95 

A A A 

4.44 
9 .06 


' 8.26 


9 . 59 


10.01 
10.02 

JLU * U / 

10.10 


2.46 
5.19 
9 .99 


6. 38 
■S.07 


l'. 5 8 
8.31 

4.72 


3.46 
9.93 


11.05 
11.06 
11.08 


6 . 03 
6.97 


3.54 
10.47 


.96 
4.77 


2.97 
8.81 


15.03 
12.04 


7,92 
3.42 


8.61 






20.01 


9.72 


' 9.^5 


11.49 














27.01 
27.02 
27.03 
27.04 
27.05 


2.01 
4.92 
1.42 
7 . 60 


3.89 
4.85 
4.80 
9.08 






28.01 
28.03 
28.02 


9.74 


2.31 

6.29 
7.44 







TABLE 



162. 



RESULTS OF RESIDUAL ANALYSIS 
BY 'site 
WRTC 









NPV 


site 


METHOD 1 


METHOD 2 


METHOD 1 


METHOD 2 


02.02 

02.09 
02.13 


2.00 
1 . 58 
1 . ."^ 1 


1.76 
2.16 
.86 




* 


03.08 
03.09 
03.16 


2.83 
2,57 
1.34 


2.73 
2.03 
1 . 29 


1.81 


2.03 


05.10 
05.11 
05.12 


2. 58 
.76 
. 35 


3. 29 
.12 
.41 






U / . 11 

07.14 
07.19 


2.76 
3 .53 


T — TC 

4.4 6 


2.64 
5.48 


2. 39 
5.29 


AO AO" 

Oo • 02 
08.04 

A A AC 

Oo • Oo 


^ 5.04 
3,64 


4 . 25 
4 1 <; 


1.02 • 
3.25 


.68 
2.93 


1/9 • 02 

A A A >l 

09.04 
09.06 
09.10 


4.31 
.64 


.15 
4.52 
.54 


.71 


.05 


10. 01 
10.02 
lU . u / 
10.10 


1 .97 

2 68 
1.49 


2.11 . 
2.30 ' 


4.67 
..3.36 
. ' -.73 


4.26 
2/38 


11 Hf^ 

13 08 


3 . 08 
2.41 


2-99 
2.66 


1.17 
2.14 


.94 


n.j3 ' 

12.04 


1.U4 
2.37 


.72 
3. 54 






20.01 


1.49 


.«6 


.08 


-.4.1 












S7.61 
27.02 
27.03 
27.04 
27.05 


2.01 
.65 
3. 22 
1.37 


1 .99 

.17 ' 
3.81 

. j9 






2d. 61 
28.03 
28.02 ^ 


. 55 
.86 
-.20 


.3D 
1.25 
.38 
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TABLE V-25 . ' 1^3. 

$ , RESULTS OF RESIDUAL ANALYSIS 
. BY SITE 

» » 

WRTD 





PV ' 


NPV 


SITE 


METHOD 1 . 


.rtETHOD 2 


METHOD 1 


METHOD 2 


02.02 
02.04 
02.09 . 
02.13 


1.12 
.65 
. 8 2 


1.16 . 

. .88 


r 




03.05 
03.08 
03 . 09 
03.16 


1.31 
.86 
1.14 . 


r. 25 
.97 
1.20 . 


.97.. 


• 1 . 0.8 


05.01 
05.10 
05.11 
05.12 


.77 
.25 ' 
. 30 


1.04.'' 
. 24' 
v26 




— 


07.11 
07.14 
07.19 


2 . 84 
2.4 7 ' 


2.69 

.:2. 71 


.90.i 
1.32 


• Q . 

1.20 


08.02 
08.04 
08.08 . 


1 . 86 
1.81 


1 .84 
2.02 


. 33 
1.09 


.17 

. .83 


09.02 
09.04 
09.06 
09.10 


.27 
1.14 

,117 


.03 
1.21 
.45 


•.46 


.46 


10.61 
1(J.02 
10.07 
10.10 


.74 

. 16 
.53 


1.09 
.61 


■ 1.25. 

■1.39 
. 70 


1.81 
.44 


11.6^ 
11.08 


.97 
. 1.43 


.97 
1 .57 


.78 
. 1.57 ■ 


'.7 3 
1 .69 


12.03 
12.04 


.96 
1 . 58 


.82 
1 .66 ' 






2o;oi 


^80 


.55 


.50 


..»6 












27.61 - 

27.02 

27.03 

27.04 

27.05 


.92 
.26 

1.02 


.73 
.13. 

"1 . 16 
' . 32 






:i8.61 

28.03 
28.02 


. 24 
- . 32 
.10 


- .06 
.02 

?I, „ 




—r 1 

• 



TABLE V-»26 
RESULTS OF RESIDUAL ANALYSIS - 



164. 



BY SITE 
ITPA 

i 





PV 


NPV 


SITE 


METHOD 1 


METHOD 2 


METHOD 1 


METHOD 2 


C2.02 
02.04 
02.09 
02.13 


2.83 

5.06 ' ' 
-2.11 


3. 77 
3.85 
1:16 




• 


03.05 
03.08 
03.09 
03.16 


-2.07 
3 .22 
.67 


6.42 ' 
1.38- 

— --O^ 


. 4.. 49 


1.68 


05.01 
05.10 
05.11 
05.12 


6.57- 
-.11 
1 .04 


2 . 55 
.?5 
-.44 






07.11 
07.14 
07.19 


2 . 52 


4. 20 

2f.00 


4.49 

-2.19 


5.91 
-2.17 


08.02 
08.04 
08.08 


.2.22 
.2.19 


2. 16 
1 .86 


1.91 

1.18 


.57 
1 .25 


09.02 • 
09.04 
09.06 
09.10 


1 .83 
-1 .05 
3.52 


1.84 
- .95 
-.06 


7.02 


f3.48 


10.61 
10.02 
10.07 
10.10 


.93 

4 .8,6 • ■ 
3 . 72 


1 . 45 
1.10 

107 


3.10 
2.08 
lOl 


2.89 
1 .36 


11.05 

IJ. . UD 

11. OP 


4 .06 ' 
2.40 


4.91 
1 .93 


5.40 
2.15. 


4 .54 

4.nn 


12.03 
12.04 


4.56 ■ 
2.57 


3.12 
..20 






20.01 


.05 


1 .69 


■ 3.94 


4.08 












TT.Ol' ■• 
27.02 
27.03 . 
27.04 
27.05 


-.8 7 
-2.31 
2.06 
. 19 


.45 
1 .08 
- .05 

.76 






2d. 61 

28.03 
28.02 ' . 











ERIC 



•a 

TABLE V-27 



165. 



RESULTS OF RESIDUAL ANALYSIS 

^ BY SITE \ . ' 

ETS 





pi 




NPV 


SITE 






METHOD 1 


METHOD 2 


02.02 
02.04 
02.09 
02.13 


2 .40 
.34 
3. 06 


2.99 
.54 
1.42 




• 


03.05 
03.08 
03.09 
03.16 


1.67 
5.37 
3.47 


1.20 
1.84 
1 .66 


1 . 39 


. 79 


05.01 
05.10 
05.11 
05.12 


5.28 
- . 35 
.99 


2.33 
.12 
.89 






07.11 
07.14 
07.19 


5 . 6S 
2 . 56 


4.04 
4. 19 


.. .00 
.91 


.- . 30 
.73 


08.02 
08.04 
08.08 


3.88 
4 . 93 


4.33 
2 .20 


1.58 
3 .71 


- 

.78 

. 56 


09.02 
09.04 
09.06 
09.10 


. 27 
- . 19 
. 28 


-.04 
1.02 
.43 


' 1 . 06 


.59 


10.01 
10.02 
10.07 
10.10 


. .43 
4.56 
1 .39 


1 .30 
.95 
1.98 


- .05 
5 .31 
1.14 


2.08 
3. 19 
fin 


11.05 
11.06 
11.08 


. 12 
1 . 35 


2.39 
1 .93 


1 .15 
M4 


2 . 77 
2 . 30 


12.03 
12.04 


1.91 
3.27 


. 58 . 
3.50 






20.01 


-1.10 


.86 




1 .54 












27.01 
27.02 
27.03 
27.04 
27.05 


-1 .40 
-1 . 25 
. 78 


2.41 
.06 
. 55 

-.10 






28.01 
28.03 
28.02 
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TABLE V-28 166. 
RESULTS OF COMBINED RESIDUAL ANALYSIS 



Program 


PS I 


PPV 


WRTC 


WRTD. 


ITPA 


ETS 


Par W^t 


2.72 


> 

5.79 


1.52 


.90 


1.83 


1.91 


Arizona 


1.61 


5.23 


2.16 


1.16 


1.81 


2.92 


Bank Street 


-.23 


5.34 


1.39 


, 50 


1.80 


1.82 


Oregon 


3.08 


7.52 


3.35 


2.67 


3.10 


3.21 


Kansad 


2.60 


6.55 


4.35 


* 1.88 


2. 11 


3. 79 


High Scope 


1.11 


3.57 


2.33 


.71 


.68 


.22 


Florida 


1.54 


.5.77 


1.93 


.67 


2.52 


1.78 


EDC 


1.89 


6.18 


2.73 


1.27 


3.11 


1.65 


Pittsburgh 


2.57 


5.80 


1.46 


1.11 


3.10 


2.52 


REC 


2.48 


9.99 


1.33, 


.70 


1.37 


.01 


En abler 8 


1.86 


4.27 


1.94 


.71 


.42 


-.19 


Control 


.23 


6.01 


.43 


.17 


* • • • 




NPV 


1.40 


6.39 


1.93 


.86 


^2.79 


1.37 


Total PV 


1.72 


5.69 


2.19 


1.06 


1.82 


1.75 


Total 


1.57 


5.89 


2.06 


.98 


2.08 


1.63 


Within Mean-square 


14.68 


64.36 


10.04 


1.47 


25.64 


10.18 


F 


9.91 


4.01 


10.91 


37.57 


2.42 


7.32 


Significance Level 


< .001 


. <.001 


<.001 


<.001 


.005 


<.00] 


* 

w 


.6 


.6 


.7 


.6 


.5 


.5 




2301 


2200 


2311 


2309 


901 


844 



167. 



TABLE V-29 



TESTS FOR S^qNIFICANT DIFFERENCES AMONG 
PV, NPV, AND CONTROL RESIDUAL MEANS* 



PV-NPV 



PV-Control 



NPV-control 





t 


Significance 


t 


Significance! t 


Signlticance 


PSI ' 


1.75 


Above 


.05 


5.66 


cpol 


• 2.79 


<.01 


PPV 


-1.77 


• 

Above 


.05 


-.40 


Above ''.05 


.43 


Abov^' .05 


vmTc' 


1.72 


Above 


.05 


4.26 


<.001 


3.63 


<.001 


WRTDj 


3.45 


^.001 




6.13 


<001 


4.60 


<001 


ITPA \ 


-3.34 


■ -.001 












1 

ETS 1 


2.11 


<.005 













* The number of degrees of freedom for each test is so large 
that we refer to noinrial distribution tables for th^ significance 
level • 
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TABLE V-30. 
COMBIKEP RESIDUAL RESULTS FOR FOUR 



SP 



168. 



PROGRAM 



Far West 

Arizona 

Bank Strett 

Oregon 

Kansas 

High/Scope 

Florida 

EDO 

Pittsburqh, 
REC 

Fnablers 
Control 

NPV 

TOTAL 

N 



P8I - 

FIRST LANGUAG12 NON IIS 
NOT KNGLISH PRIOR PS 



2.11 
.73 

3.49 ** 

1.57 
2.85 



4.81 ** 
1.44 
.91 

-.14 

1.92 

161 



V 



.93 
-.81 
-3.54 
-.55 

-1.66 
.02 
2.91 



2.85 
-.58 

.33 

.17 

133 



* Indicates significance p. 05 
**Indicates significance p .01 
***Indicates significance p .001 



NO^PRIOR 
PS 



PRIOR HS 



2.83 
2.22 
1.36 
3.66 
3.02 
1.27 
1.60 
1.55 
3.05 
2.17 
1.79 
.50 

1.87 

1.90 

1577 



*** 

** 



.65 
-.74 

.86 
•2.77 
1.34 

.20 
2.18 
1.49 

.57 



,68 
.64 
430 



Significance; is foi conl-.r?.G;;.'..'i lii 1,1'V 
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TABLE V- 31 

COMBINED RESIDUAL RESULTS FOR FOUR 
GROUPS ON WHICH REGRESSIONS ARE BASED 
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PROGRAM 



FIRST LANGUAGE 
NOT ENGLISH 



V 



NON HS 
PRIOR PS 



NO PRIOR 
PS 



PRIOR US 



Par West 

Arizona 

Bank Street 

Oregon 

Kansas 

High/Scope 

Florida 

EDC 

Pittsburgh 
REG 

Enablers 
Control 

NPV 

TOTAL 

N 



8.17 
4.65 

8.71 * 

14.50 
16.16 
-3.30 * 

17.10 
6.30 
3.45 * 

13.30 

10.77 

150 



-2.21 
1.07 
1.38 
■13.09 

-.68 
1.46 
15.78 
2.97 
5.56 
1.11 
3.40 

.68 

1.91 

133 



6.03 
5.64 
6.32 
7.45 
6.82 
3.14 
5.47 
5.40 
6.39 
8.90 
4.81 
7.40 

6.13 

5.89 

1518 



* 



3.88 
5.05 
4.88 
5.05 
3.73 
4.01 
2.06 
5.08 
4.84 

-.16 



5.88 
5.36 
399 



Indicates sign 
Indicates sign 
Indr'.cates sign 



if icance p . 05 
if icance p . 01 
ificnncG p .001 
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/ TABLE V- 32 

COMBINED RESIDUAL RESULTS FOR FOUR 
GROUPS ON WHICH REGRESSIONS ARE BASED 
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PROGRAM 



Par West 
Arizona 
Bank Street 
Oregon 
Kansas 
High/Scope 
Florida 
EDG ' 

Pittsburgh, 
REG 

Enablorr, 
Control 

NPV 

TOTAL 

N 



WRtC 

riRST LANGUAGE NON HS 
NOT ENGLISH PJ^IOR PS 



2.60 
2.26 

3.77 

.93 
2.29 
•2.51 

2.49 
1.88 
3.88 

4.82 

3.53 
173 



* 

* 
* 



* Indicates significance p .05 
♦♦Indicates significance p .01 
***Indicates significance p .001 



.19 
1.19 
.48 

rl.38 

-.50 
2.21 
6.01 
1.03 
.97 
3.05 
-.56 

.53 

.99 

123 



NO'^PRIOR 
PS 



1.52 
2.02 
1.25 



4.53 
4.53 
,2. 54 
1.82 
3.21 
1.64 
1.14 
1.78 
\35 



1.68 
1.99 
1576 



*** 
*** 



** 



PRIOR HS 



2.04 
2.69 
1.66 
2.85 
2.29 
3.90 
2.43 
2.18 
.29 

1.79 



1.71 
2.02 
439 



y 



•Signx r.ioonoc.' xr. for contj'. 



■/If :i 
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TABLE 3 
COMBJCNED RESIDUAL RESULTS FOR FOUR 
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PROGRAM 



Par West 

Arizona 

Dank Street 

Oregon 

Kansas 

High/Scope 

Florida 

EDC 

Pittsburgh 
REC 

Encjblers 
Control 

NPV 

TOTAL 

N 



WRTtf 



FIRST LANGUAGE 
NOT ENGLISH 



2 . 33 






.57 




.74 






. 51 


2.78 


*** 


.42 


.41 


* 


.35 


.02 




.08 


.54 




1.71 






.44 


1.76 




.36 


.56 


* 


.78 


.85 




.11 


1.29 




.84 


1.60 




.58 


173 




123 



* Indicates significance p .05 
♦♦Indicates significance p ,01 
***Indicates significance r> .001 



NON HS 
PRIOR PS 



NO PRIOR 
PS 



.90 
1.17 
.41 
2.77 
1.93 
.70 
.83 
1.02 
1.21 
.52 
,74 
.12 

.77 

.92 

1576 



*** 
*** 



PRIOR HS 



1.01 
1.25 

.63 
•2.20 
1.32 
1.54 

.05 
1 . 52 
1.08 

-.08 



.'94 
1/.06 
437 



Siv^jnif i cahcG i for coi: L rv^l; \:ilh UVV 



TABLE,V-3 4 
C OMBINED RESIDUAL RESULTS FOR FOUR 

omH 6K^ifatdi itE6Rte^616tT6 Ate feAl'ED 
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PROGRAM 



Far West 
Arizona , 
Bank Street 
Oregon 
Kansas 
High/Scope 
Florida 
EDC 

Pittsburgh 
REC 

Enablers 
Control 

NPV 

TOTAL 

N 



ITPA 

FIRST LANGUAGE NON HS 
NOT ENGLISH PRIOR PS 



•2.52 

2.43 * 

2.48 
3.94 



.39 
■4.21 



•1.41 
.57 

68 



'2.28 
3.10 
3.02 



.98 

3.32 
•2.56 
1.17 
.72 



4.65 
.84 

47 



NO PRIOR 
PS 



1.96 
1.64 
1.49 
4.21 
2.10 
.73 
.73 
3.57 
3.87 
1.80 
.69 



3.05 
2.17 
611 



PRIOR Ili-'i 



3.58 
2.25 
2.13 
2.76 
2.17 
•3.24 
•3.24 
2.63 
5.72 

-.72 



3.50 
2.70 
175 



* Indi.cates significance p .05 
** Indicates significance p .01 
***Ind:'.cates significance p .001 



Siyr i j'icanco i.s 'ov vcv.trc.^jl ./.Ll i 'AW 



TABLE V-3 5 

COHBINED RESIDUAL RESULTS FOR FOUR 
GROUPS ON WHICH REGRESSIONS ARE BASED 
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PROGRAM 



Far V^est 

Arizona 

Bank Street 

Oregon 

Kansas 

High/Scope 

Florida 

EDC 

Pittsburgh, 
REC 

Enablers 
Control 

NPV 

TOTAL 

N 



ET^ 

FIRST LANGUAGE 
NOT ENGLISH 



NON HS , 
PRIOR PS 



■2.03 

3.11 ** 

■1.58 
3.53 
■3.35 

1.13 
-.80 

.49 

137 , 
67 



1.76 
2.43 
■2.67 



.48 

.68 
3.64 

.00 
-.95 



1.00 
.76 

43 



* Indicates significance p .05 

♦♦Indicates significance p .01 

***Indicates significance p .001 

.GigniCicancc !.:• for con tra .,iLl: ;;: v 



tTO'^PRIOR 
PS 



1.79 
3.53 
1.94 
3.05 
3.85 
.30 
1.44 
1.77 
2.42 
-.29 
».03 



1.55 
1.67 
565 



PRIOR HS 



3.15 
1.73 
2.03 
3.89 
3.40 
1.48 
3.09 
1.74 
.75 

-1.53 



1.13 
1.79 

169 
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Table, v-36 



Total Sanple Size for Residual Analysis 



- Teat Valid Fall Valiu Spring Msthod 1 Method 2 Corbined 



PSI 


3175 


2753 


2302 


2460 


2301 


PPV 


3217 


2660 


2343 


2367 


2200 


WRIC 


3204 


2792 ' 


2311 


2490 


2311 


WCTD 


3204 


2792 


' 2326 


2488 


2309 


ITPA 


1210 


1077 


960 


976 


901 


ETS 


U35 


2606 


847 


^328 


844 
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One final remark concerns the efaiiiples on which the 
various analyses were based. The data requirements for' 
Methods 1 and 2 differ slightly* and those for the Com- 
bined analysis are the most stringent. We would be some- 
what concerned if the data collection did not allow com- 
putation of residuals for very many children. The sample 
sizes summarized in Table V-3 6 would seem to reveal no 
cause for concern. 

Results by Test 

In this section we present a summary of the results 
of the residual analysis for each test. These svunmaries 
are based primarily on the combined residuals. 

Preschool Inventory 

The average expected increment for all PV children 
was 3.07, for NPV children 3.01, and for Control children 
2.51. The average residuals were 1.72 for PV/ 1.40 for 
NPV and .23 for Control. Thus the growth rate for Head Start 
(PV and NPV) children over the period between tests in- 
creased by roughly 50%, while the rate for Control 
children increased negli^iRly. Putting it another 



*The main difference is that Method 1 requires fall scores, 

while Method 2 does not. * 

» ] 

■ " - A 

o 
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way, we would have expected Head Start children to gain 
about #5 of a standard deviation {see Table XI-25) without: 
any preschool; with Head Start they gained about .75 of 
a standard deviation. The difference between PV and NPV 
was not significant at the .05 level. The overall F-test 
for program differences was highly significant. The mean 
residuals for Oregon (3.08), Far West (2.72)/ Kansas (2.60)^ 
and Pittsburgh (2.57), and B£C (2.48) were high. Bank 
Street (-.23) and the Controls (.23) were low. Most of 
these effects seem fairly consistent across sites, but 
Kansas is rather puzzling. The Portageville Bite showed 
the highest mean residuals for the two methods (5.02, 
5.48), while Mounds did very poorly (-1.51, 1.06). On 
.the whole, children without prior pre-school experience 
had larger residuals than those with prior pre^school. 

Peabody Picture Vocabulary Test 

The average expected iij^rement for all PV children 
was 5.04, for NPV children 4.76, and for Controls 4.70. 
The average residyals were 5.69 for PV, 6.39 for NPV, 
and 6.01 for Control Thus the growth rate for all three 
groups more than d<^ilbled* In terms of standard deviations, 
th(ft expected growth was about .35 and the actual growth 
about .8 for all three groups. The differences among the 
three groups were riot si.gnifiqant at the .05 level. 
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The overall P-test for program dJ.f iferences was significant 
(p <.001)* The mean residuals for REC (9*99) and Oregon 
' .\ (7.52) were high, and those of High/Scope (3.57) and the 

« 

EnablejP^(4.27) low. There appear, however, to be large 

*/ 

variations among sites within models* The 150 children 
' whose first l^ifguage was not English and who were in a * 
Head Start prograin had an average residual of 11,58, 
while the 7 .in ihe Control group averaged only 3 •45, 

WRaT ^Copying Mark|g > ^ - ^ ^ 

' The average expected increment for all Py children 

was 1,15 1 for NPV children 1,11, ^and for Control children 
1«09« The iaverage residuals were 2,19, 1,93, and .43 
respep^jlvely. Thus, while the growth ^aten for PV and |JPV 
chiiaren. nearly tripled, we must remember that^ since the 
mean fall score was only 2^03 On a test with a maximum of 
18, the spring mean of 5,33 was still rather loU. In terms 
of standard deviations, the expected gain was about ,4 
and the actual gain a^out .75, The PV and NPV means did 
not; differ significantly, but both'we re significantly (p<,001) 
above the Control mean. The overall F-test for program 
difference w^^highly significant. The mean residual for 
Kansas (4 .35) and Oregon (3.35) stood out on the high side^ 
while the Controls (.43) were by far the lowest. These 
results were consistent across all sites within these ms'^dels, 
although other models (most notably Bank St:reet 

ERIC 
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4 

and High/Scope) showed large site to site variations. 

WRAT Reading Numbers 

The average expected increment for all PV children 
was .32| tor. NPV children .29 and for Control children .25.' 
The average residuals wexe 1.06, .^6, and .17 respectively. 
Thus the growth rates for PV and NPV quadrupled ^ Of 
course the projected growth rate was rather small. In 
terms of standard deviations , the expe M ed gain was about 
.3 and the actual gain over 1.0. The PV mean was signifi- 
cantly. (p < .001) higher than the NPV mean, but this was 
probably attributable to two outstanding PV models (Oregon ^ 
and Kansas) . Both PV and NPV were significantly (p < .001) 
above the Cofitrols. The overall F-test for program differ- 

r 

ences was highly significant. Oregon (2.67) and Kansas (1.88) 
clearly stood out on the high side. The Cbntrols (.17) 
and Bank Street (.50) wereiow. Results seem quite consis- 
t^ent across sites. 

ITPA Verbal Expression . 

The average expected increment fo^r all PV children was 
1.89; and for I^PV children 1.65.. Controls were not given 
the ITPA. The average residuals were i.82 for PV >nd 2.79 
forl^PV. Thup the griDWth rate for Head Start children more 
thai) ^doub led. ^ I^n terms of standard deviations, the expected 
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gain was about .35 and the actual gain about .1 for PV and 
.85 for NPV. The difference between PV and NPV was sig- 
nificant (p<.OOI) and somewhat perplexing. The overall 
P-'test for program differences was significant {p«.005) . 
EDC .{3.1 if, Oregon (3.10) # and P^-ttsburgh (3.10) had the 
highest mean residuals, while Enablers (.42) and High/ 
Scope (.68) were lowest^. The results' seem fairly consistent 
across sites. 

ETS Enumeration. 

The average expected increment for all PV children 
was 2.02 and for NPV children 1.82. Controls were not 
given the ETS. The average residuals were 1.75 for PV 
and 1.37 for NPV. Thus the growth rate increased by about 
75%. l^n terms of standard deviations, the expected gain 
was about .4 and the actual gain about .7. The difference 
between PV and NPV was barely significant at the .05 level. 
The overall P-test for model differences was significant 
(p<.001). Kansas (3.79), Oregon (3.21), Arizona (2.92), 
and Pittsburgh (2.52) wterehigh. Enablers (-.IP), REC ^ 
(.01), and High/Scope (.22) were low. Effects seem fairly 
consistent across sites. 



ERLC 

I k 



180. 



Summary of Residual Analysis Reeults 

As we did for the ranging analysis, we shall summarize 
In this section the evidence provided by the residual 
emalysls bearing on our three major questions*. 

It To wHat extent does a Head Start experience 
accelerate the rate at which disadvantaged 
pre-schoolars acquire cognitive skills? 

» 

Our evidence here Is clear and direct. Children In 
Head Start programs apparently gained substantially more 
on each ttBt than they would have without the programs. 
For all tests except the PPV, the Control children showed 
small average residual gains* Since there is bound to be 
some test sensitization or slight imperfection in our 
regression models # these results are quite consistent with 
what we might expect, and further evidence thai: the Increase 
in growth rates for Head Start children are genuine program 
effects. and not mathematical artifacts. We do not, however, 
understand why the Control children 6n the PPV showed an 
increase comparable to that of the Head Start children. 
From this and the ranking analysis it seems clear that the 
Controls performed about as well as the Head Start children 
on the PPV. The question is whether Head Start progrfuns 
really have no effect, so that , the residuals are some 
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kind of artifact, or whether for some reason both Head Start 
and Control children do better than we would expect pn the 
basis of natural maturation. We shall explore this 
perplexing issue somewhat further in Appendix F.. A 
particul^irly interesting finding about the PPV was- the 
tremendous ihcrease in scores for children from Spanish 
speaking families. Head Start may be functioning for these 
children as an effective early exposure to the English 
la^nguage. This effect seems to hold only for preceptive and 
not active vocabular^fj as the residuals of Spanish speaking 
children on the ITPA were rather low. 

2. Are the Planned Variation* models simply by 
virtue of sponsorship/ mor6 effective than 
or^ln^ry, non-sponsored Head Start programs? 

On three of the six tests (PSI, PPV, WRTC)^ the ' 
difference between PV and NPV mean residuals falls to reach 
significance at«the .05 level. The difference for the ETS 
is barely significant at the .05 level. E!or the WRTD the 
PV mean is significantly (p < .001) higher^ and for the ITPA 
the NPV mean is significantly (p < .001) higher. The 
difference for the WRTD can be primarily attributed to 
the stai>d-out performance of two models (Oregon and 
Kansas) • The ITPA difference seems attributable primarily 
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to two models (High/Scope and Enablers) which stood out 
negatively • Our Impression Is that, on the whole, the 
performance of PV and NPV programs Is quite comparable. 

I 

3. Are some PV models particularly effective at 
Imparting certain skills? 

Table V-37 presents a summary based on the discussion 
In the previous section. Of the 22 "effects'^ noted, 16 
occur In 3 of the 6 tests (PSI, ITPA, ETS) . The only test 
with fewer than 3 effects is the PPV, which has none* As 
In the ranking analysis. It appears that the PPV Is not 
particularly sensitive to program differences. In terms of 
models. It Is Interesting to note that of the 15 positive 
effects, 11 are for the "academic** models (Oregon, Kansas, 
and Pittsburgh). Moreover, all 4 4-i-*s are for tliese models. 
Thus, as in the ranking analysis, the evidence suggests 
that i:he academic models may be generally more effective 
in transmitting academic skills. 
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Table V-37 

Summary of Relative Model Effectiveness 
Based on Residual Analysis ^ 



++ Indicates model appears to be highly effective. 

+ Indicates ^videpce for above average effectiveness. 

- Indicates evidence for below average effectiveness. 

— Indicates model appears to be highly ineffective. 



Model 


PSI 


PPV 


WRTC 


WRTD . 


ITPA 


ETS 


Far West 














Arizpna 












+ 


Bank Street 














Oregon 






+ 






4+ 


Kansas 














High /Scope 














Florida 














BDC 














Pittsburgh 










+ 




Enablers 















*REC not included because with only^one site we felt it 
xanfair to draw ahy conclusions. 
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Chapter VI 
ANALYSIS OF COVARIANCE 

Theory of the Analysis of Covariance i 

In this section we discuss the theory underlying what 
is currently perhaps the most popular technique for com- 
paring the effects of educational programs in quasi- 
experiiaental situations, the analysis of covariance (ANCOVA) • 
We begin with the more general problem of constructing 
linear models to describe the relationship between post-test 
scores and variables which can be measured prior to program 
exposure, including the pre-test score* Let us, for con- 
venience, refer to all such preprogram variables as co- 
.^variates. Suppose for each program we could fit a regression 
model which would allow perfect prediction of a child's 
post-test score on the basis of the available covariates. 
Then, in theory at lea^t, we could compare the effects of 
different programs on children with any specified set of 
background characteristics. > In practice, wo can predict 
with only limited accuracy. Moreover, there would be a 
virtually infinite number of possible comparisons, one 
for each possible combination of child background char- 
acteristics. To summarize all this information in a 
meaningful way would be quite difficult • 
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Sup{$bse, however, it turns out that a simpler mathe- 
matical model is adequate. Suppose that the post-test 
score Y! for any child can be predicted by some function 
(Sv^y P) of his covariate values (say V) plus an additional 
effect attributable to the particular program experienced. 
Thus, for individual i in program j we would ha,ve 

where €tj ^presents a program effect and e^^ random error 
uncorrelated with the covariates* If F is a linear function 
of the covariates, it can be separated into a part involving 
the pre-test Y and a remaindeir, say involving the othjsr 
covariates. Thus we have 

V 

Y.j- = + gif. . + M^j + e^^ (6.2) 

If this model is appropriate, it provides straightforward 
treatment comparisons. We simply fit the model and compare 
the values of the effects ttj estinated for the various programs. 
Each a. may be considered as the expected value for 
individuals in program 3 after "adjustment" for the pre- 
test and other covarj tes* 

Note that the assumption that the function F (i.e. the 
set of regression coefficients for the covariiites) is the 
same for all progrcun groups is absolutely essential in 
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allowing straightforward program comparisons. To clarify 
this point, .suppose for the moment we have only one covar- 
iate, the pre-test^ and are comparing two programs. Then 
over the range of possible pre-test scores, there are 
essentially three possibilities, as illustrated in Figure 
VI-1. In situation (a) we cannot say which jirogram is . 
better. For children with low pre-tests program 2 is better. 
For those with high pre-tests 1 is better. In (b) we 
can say that 2 is generally better, but we haye no simple 
measure of its superiority, since the difference between 
the program effects varies with pre-test score. Only in 
situation (c) can we say simply that program 2 is on the 
average 0^2 ^1 points better. 

With more than one covariate the situation becomes 
more complex. The assumption that F is the same for all 
program groups becomes more difficult to check. 

If the ANCOVA model is basically correct r the precision 
of group comparisons based on it depends on how much of 
the within-group variance can be explained by the covariates^ 
As explained in Chapter V, the reliability p of the post- 

test is an upper bound on the proportion of variance 

' • ' 2 

explainable. Thus, our goal is to. build models with R as 

close to p as possible. 

A rather thorny issue which is the focus of much 

current concern involves effects on the ANCOVA of unrelia- 
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Figure VI-1 

Possible Relationships between Pre- and Post- 
Tests for Two Programs 



Post-Test 

Program 2 
Program 1 



Post-Teat 

Program J 
Program i 



Post-Tes 



tt 



Program 2 
Program 1 




Pre-Test 



Pre-Test-^ 




} ^ - ^ 



Pre -Test ~> 



(a) 



(b) 



(c) 
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bility in the covariates. If the model described by 
equation (6.2) is correct, there Is no theoretical problem* 
Some researchers feel, however, that a linear model stated 
in terms of ''true scores" rather than observed scores is 
more appropriate. Suppose that 

where j and T^^ ' are the, true scorefs corjpesponding to Y^^ j 
and j ■ respectively, and Uj^j and j ' are random errors 
of measurement with mean 0 and uncorrelated ^^ith the true 
scores* For simplicity, suppose the pre-test is the only 
covarlate* Then we can consider a mathematical model of the 
form 

T^j' = a. + bT. . (6-3) 
which implies ^ 

Y.j- = a.+ bY. . + {u.^» - ^^ij^- ^^'^^ 

In this model, the error term is correlated with the pre- 
test, a violation of the usual assumptions oh the basis 
of which linear models are fit. If we try to estimate the a^ 
and b using the usual least-squares procedure, we obtain 
biased estimates. 
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\^ As we mentioned in Chapter IIIi several suggestions 
for Correcting" the ANCfOVA for unreliability of the co- 
variates have recently appeared (e.g* Lord, 1960; Porter, 
1971) • These corrections seem to us rather shaky for use 
at the present time in educational evaluations. For 
one thing, they depend heavily on the rather stringent 
classical assxxmptions about errorf of measurement » Second, 
they make the somewhat arbitrary assumption that a linear 
model holds in terms of true scores but not- observed 
scores. Third, they require a fairly precise knowledge of 
*the covariate reliabilities, and finally, from a practical 

•a . - , " 

standpoint they are difficult to implement, particularly J,n 
the multiple cova :iate situation. It seems to us more ^ 
fruitful to try to explain as much variance as possible 
using a limited number"^ of reasonably reliable covariates. 
In concluding this section, let us summarize the main 
/ assumptions on which the use of ANCOVA is based. 

AssumptiOR!^ I s A linear model adequately represents 
the relationship between post-test and covariates for 
each program gifoup. 

A saumrhlon 2 ; The coefficients of the covariates In 
the different program groups are approximately equal. 

Assumption 3 : The covariates have high enough relia- 
bility to avoid serLousiy biased results. 
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Implementation of the Analysis of Covariance 

Several exploratory regression analyses were carried 
out with the^sprlng test score (post-test) as the dependent 
variable and a variety of covarlates, Including the pre- 
test score and fall scores for other tests. For practical 
reasohsr we limited these preliminary investigations to 
three tests (PSI^ PPVr WRTC) . 

Although we felt that interpretation would be easier^ 
if we could avoid Interaction variables, it became clear 
that iijt^raot^ involving child ethnicity could not be 
ignored* Since we wished to avoid the Intr Auction of 
many two-way and even higher order interactlonor we felt it 
would be simpler to divide the sample into' Blacks, Whites, 
and Spanish Americans (Mexican Americans and Puerto Rlcans) 
and to build separate regression models* Within these 
ethnic groups/ separate regression models were fitted for 
each of the thirteen prv^ram groups, (11 PV models, NPV, 
Controls) with a sufficient number of children*. One 
of the most promising sets of models is displayed as an 
example in Tables VI-1 through VI-9. 



*One model (Pittsburgh) contained no Blacks, and 5 (Par 
West, Arizona, Kansa, EDC, Pittsburgh) had not ehough 
Spanish Americans to carry out the analysis* 
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Looking oyer the various equations^ we noticed that 
a few variables tended to predominate in importance. These 
were the pre-test scored fall scores for som'e other tests, 
and age. Using only these variables, we fit the equations 
displayed in Table VI-10 through VI-18. Note that generally^ 
there is only a small loss in compared with the more 
complex models described above« Moreover, except possibly 
for age, the coefficients seemed fatirly constant 'across 
the different models within any ^thnic group. We decided 
to perform ANCOVA's for each of the 8 tests for Blacks* and 
Whites. We felt there were not enough models with a sub- 
stantial number of Spianish Americans to justify running 
the AMCOVA for them. We also decided to eliminate age as 
a covariate. We ended lip using as covariates the pre~test 
score and fall scores for the PSI and PPV. We considered 
the possibility of carrying out a formal statistical test 
of the assumption that the regression coefficients for 
different groups were the same. An attempt to do this would, 
however, have involved us in computational problems 
beyond the capabilities of; the computer programs available 
to us. 

The ANCQVA wds carried out using a standard multiple 
regression program*. For Whites there were 13 treatment 
groups {11 models, NPV, Controls) and for Blacks, 12 



Table VI -16 



REQRBSSION MODEIiS RELATING PSI POST-TEST TO 
PRE^TEST/ other fall test scores', At<D AGfi . 



vmites 
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Model 


0 


PSI 
Fall ' 


ppv; , 

Fall' 


4« 

Age 




r2^ . 


Far West 


6.47 


.54r 


.074 ^ 


.061- 


137 


.554 ^ 


Arizona 


6.71 


.463 


•.114 


.046^ 


125 


.543 


Bank Street 
Oregon 


3.97 


.574 


.123 


.034 


41 


■ ;787 

J- 


Kansas 


-15.3 


.359 . 


.145 


' . '463 


i . 

107 


<»574 


HjLgh/Scope 


2.20 


.516 


.077 . 


.114'-" 


/'.643 


Florida 


-15.2 


.716 


-.146- : 


.448 


39 


i552 


EDC 


19.2 


.638 


.080 / 


-.159 


39 


.628 . 


Pittsburgh 


7.66 


.513 


.146 


,002 


119 


• .656 


REC 














Enablers 


3.56'| 


.499 


.144 


.062 

f. . 


103 ^; 


.768 . 


Control 


-3.77 1 


.644 


,145 


.099 . 


. 54 ' 


.747 


HPV 


2.68 


.494 


.128 


.090 


205 . 


^623;.' 



ERIC 



Table ^^I-ll^ . . y S:'''"-^:-- 
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REGRfi$8iON MODELS RELATING PSI POST-TgST TO 

* ■ ' , 

PRE^TEST, OTHElR FALL TEST SCORES, AND AGE 

Blacks 

• • ■ 
# ■ - _ • 



^ Model 


c 


P<5t 

XrOX 

Fall 


Fall 


,, Age 


n 




Par West ' 














Arizona . " 


1.86 


%746, 


.008 


.096 


74 , 


.673 


Bank Street 


,963 


.463 


.125 


.095 


215 


.724 


Oregon , 


13.2 


.423 


.145 


-.038 


74 


• 551 


Kfinsas . ^ 


-9.21 


.547 


.119 


-.287 


61 


~T4i6 


High/Scope 


-.815 


.071 


.294 


.146 


54 


.411 


Plorida 


5.64 


.518 


.141 


.016 


103 


.508 


EDC 


-3.16 


.417 


.064 


.226 


129 


,553. r 


Pittsburgh v 








n 






REC 














-Enablers . 


3.86 


.414 


• .139. 


.048 


74 


.411 


Control 


-3.62 


.629 


.205 


.088 


sr 


7.37 


NPV 


1.55 


.543 


.104 


.097 


383 


. 599 



ERIC 
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, REGRESSION MODELS RELATING Hi P0ST-TE8T TO 
. PRE-TEST, OTHER FALL TEST SCORES, AND AGE 
Spanish Americans 



203. 



41 



Model 


o 


PS I 

^Eall 


t>PV 
Pall 


Age 


n 


r2 


Par West 














Arizona 












- 


Bank Street 




f 


«>.- 








Oregon 


-1.32 


' .417 


.047 


.246 


76 


. .675 


Kan&^Qs 














High/$cope 


17.97 


i482 


.139 


-.196 


33 


.335 . 




-.324 


.386 


.203 


.130 


30 


.354 


EDC 














Pittaburgh 














REC 


15.6 


.561 


..130 

1 


-.169 


44 


.507 


Enablers 


-4.07 


.423 


.087 


.239 


47 


.640 


Control 


—2.60 


.670 


.117 ' 


.108 


21 


.829 


NPV 


-2.65 


.518 


.101 


.189 


111 


.623 



erJc 
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« I 



MGRESSION MODELS RRIATIMfl ppv pqst-TEST TO 
PRE-TEflTf OTMRR FALL TEST SCOREfi ^ AMn 

Whites , - 



204. 



Model 


0 


Pall 


PPV 
Pall 




1 


~3 


Par West 


29.9 


.^9 




Ago 

' A o e 


131 


.550 


Arizona . 


28.8 


. 322 


A O f\ 


-•072 


121 


.484 


Bank Street 


19.9 


.335 


• ^ / D 


« 043 


" 41 


■ .588 


Oregon 






\ 








Kansas 


13. ,4 


.086 


.564 


• 202 


42 


.574 


High/Scope 


13.1 


.310 


.582 


.093 


103 


.677 


Plorlda " 


.* Ct AC 

- • o 4 5 


-.191 


' .788 


.304 


39 


.703 


EDC 


32.4 


.272 


.428 


0fi2 




• 537 


Pittsburgh 


26.4 


.214 


.490 


.016 


115 


.611 


RBC 














Enablers 


20.9' 


.406 

f 


.^96 


.009 


100 


.665 


Control 


1.97 


.550 


.503 


.299 


48 


.718 


NPV 


18.1 


.320 


.476 


.104 


204 


.561 
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R EGRESSION MODELS RELATING PPV POBT-TEST TO 
PRE-TEST> OTHER-FALL TEST SCORES » AND AGE 

Blacks 

■ •" "' : ' ' ' , *!?. 

/. ■ . ^ * ■ * . ' 



Model • 


0 


FSI 

Fall 


rrV 

Pall 


Age 


n 


r2 


Far West 


■ 












Arizona 


5.84 


.824 


.627 


.066 


74 


.582 


Bank Street 


12.1 


.307 


.594 


.067 


221 


. .631 


Oregon 


37,2 


.694 


.346 


-.237 


61 


.675 


Kansas 


15.3 


.249 


.478 


.111 


61 


.438 


High/Spope 


-1.22 


-.231 


.962 


.250 


53 


.691 


Florida 


10.2 


.292 


.452 


.191 


93 


.439 


EDC 


.139 


.216 


.646 


.272 


126 


.616 


Pittsburgh 














REC 














Enablers 
Control 


.738 


.327 


.576 


.267 


73 


.481 


5.05 


.290 


.600 


.214 


52 


. 574 


NPV 


10.4 


.397 


.509 


.133 


365 


.546 



ERJ.C 
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REGRESSION MODELS RELAThIq PPV POST-TEOT TO 
PRE-TESTt OTHER FAJ.L TT^ST f^CORRS/ AND AGE 

Spanish yyrier loans 



Model 


o 


P6I 
Fall 


/ PPV 
1 Fall 


Acfe 






Far West 




t 

t 










Arizona 










< 

- 




Bank Street 




i 

^ 1 










Oregon 


-12.3 


.097 ! 


.561 


.570 


74 


.616 


Kansas 




(. 












40i 6 


"•462| 


.627 


-.368 


32 


.369 


.Florida 


-32.7 


X.2B 1 


.411 


.756. 


29 


.610 


EDC 




i 








■ 


Plttitburgh 














REC 

* 


5o;o 


.ije ■ 


.696 


r.531 


41 


.599 


Enablers 


5.13 


.3pO 


.533 


.272 


41 


i697 


Conttcrol 


22.5 


(. 

-.^10 


.814 


-.137 


19 


.694 


NPV 


7.82 


.533 


.421 


.243 


111 


.380 



Table Vl-16 



REQRESSIOK HODEIiS RELATINQ WRTC POST-'TEflT TO ' 207; 
>RE--gEST, ,OTHBIl F'ALL TEPT SCORF.S , km AGE 



|5r Meet 

ktik street 
fegon 
neas 

.^ih/Scope 
or Ida 

ttaburgh 
C 

ablers 
>Atrol 



-.92 
-2*63 
-5.31 

-9.64 
■17.7 
-19 . 8 
.499 

-4.22 

\ ' ; 

-2.27 
-2.93 

-9.34. 



Whites 



PSI 
Pall 



.147 

.203 
.316 

.060 
.099 
.159 
. 431 
.094 

.168 
.147 
.223 



Fall 



.015. 

.011 

.026 

.073 
.036 
-.174 
-.106 
.060 

.010 
.061 
..03 a 



Age 



.040- 
.050 

.on. I 

.220 
.334 
.418 
.0^8 
.086 

^.043 
.024 
.147 



WRTC 
Pall 



.544' 

.462 

.i23 

.293 
.418 

.112 
.603 

.817 
.205 



13 A 
129 
43 

42 
110 
40 
38 
118 

106 

40 

Ji-13 



.:?6? 
.47? 
.545\ 

.295 
.641 
.605 
.500 
.479 

.072 
.668 
.505 



;l 



■ ' ■ . ■table Vi-17 . ^ ■ 

REGRESSION MODELS RELATING WRTC POST->TEST TO 
PRE'TEST, OTHER RALL TEST SCORES ^ AND A GE 



-3.21 

-5.06 
16.3 
-3.54 
4.41 
-6.66 
.922 



2.75 



.324 



PSI 
Pall 



.042 

.134 
.011 
-.048 
.183 
.238 
.027 



Blacks 

PPV 
Pall 



010 



.031 



.038 

.010 
.116 
,124 
.031 
.016 
.026 



.079 



-.005 



Age 



.088 

.100 
.239 
.117 
.115 
.094 
.048 



-.049 



<.O04 



WRTC 
Fall 



.623 

.633 
.599 
.644 
1;13 
.570 
.427 



.937 



.710 



n 



77 



228 
71 
66 
52 
99 

134 



78 



41 



-4.86 



.036 



.053 



.084 



.887 



396 



II 
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RRfiRRflSIQN MODELS R ELATING WRTG POST-TEST TO 
PRE-TEST, OTHER FALL TEST SCORES, AND AGE 

Spanish Americans 





0 


PSI 
Fall 


:ppv 

Fall 


Age 


WRTC 
Pall 


n 


R 


' ' ' 
IT West 
















izona 
















nk Street 


















19.2. 


.126 


-.076 


-.182 


.629 


77 


.359 




■ ' c 














gh/Scope 


9.30 


^ .218^ 


• T.148 


-.091 


.865 


34 


.463 


orlda 


.926 


-.035 


.197 


.014 


.264 


27 


.417 


9 .. . ' . . . 










i 








2.30. 


,214 


.057 


r.063 


.773 


45 


.326 




.626 


.031 


.053 


.023 


.814 


53 


.383 


ntrol 


i • 
-11.9 


-.251 


.081 


.285 


.816 

; f * 


18 


.570 


- 


-24.2 

■ \ 

\ 


.205 


-.022 


.460 


.345 


117 


.636 


^ ■ • 

ERLC 

















1 
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treatment groups (10 models, NPV, controls). In each case, 
we could include as independent variables along with the 
covaria^tesi dummy variables for all but one of the treat- 
ment gxoups. The coefficient' of a dummy variable estimates 
the difference in program effects (aj*s) between the corres- 
ponding treatnjent group and the "base" group for which.no 
dummy was included. In our first runs, we used the Control 
children as the base group for our cpmparisonst The results 
of these analyses appear in Tables VI -19 through VI-24* 
Since wo were also interested in the signiAcance of 
comparisons between PV and NPV, we ran another ANCOVA with 
the Controls deleted and the NPV children as our base. 
The results appear in Tables VI-25 though VI-32. 

Results of the Analysis of Cevariancfe by Test 

In this section we present brief summaries of the 
ANCOVA results for each of the 8 tests in our battery* In 
this section, when we refer to an "effect" of a program, 
we mean the estimated dlfference_betwee^ and 
that of the Controls. When we say simply that an effect 
is significant, we mean at least at the .05 level. 

" I 

*Thi8 could not be done for the ITPA and ETS s;".nce the 
Controls were not given these test4. 

ERIC ' ' 



Table VI -19 



RESULTS OF ANALYSIS OF COVARIANCE 
(EFFECTS RELATIVE TO CONTROLS)* ,. 



211. 



PSI 
White 



Black 





Effect 


t 


Effect 






4 63 




4 . 24 


4 . 60 






7.67 


3 41 


5.03 




2.90 


4.11 


1.80 


3.09 


Oregon 


5 94 


5.77 


5.86 


8,52 


l\anSaS 






9 7fi 
• / o 


Q5 


nxgn/oCopG 




R fin 


2 2 5 


2 09 


Florida 


2 ,94 


4.07 


3.29 


5.15 


EDC 


5.28 


7.33 


3.84 


6.22 


Pittsburgh 


4.79 


8.58 






REC 


2.27 


2.29 


2.97 


3.16 


Enablers 


4.14 


7.15 


, 1.97 ' 


2.91 


NPV 


4.09 


7.84 


3.37 


6.04 


Control 










Constant 
Fall PSI Coeff. 
Fall PPV Coeff. 
Pall Coeff. 

n 


3 . 29 
.543 
.125 

152.39 
.676 

1039 


3.23 
.550 
.127 

152.99 
. .613 
1268 



t > 1.96 is equivalent to p < .05 
t > 2.58 is equivalent to p < .01 
t > 3.27 is equivalent to p < .001 
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■- 'table vi-2b 

RESULTS OF ANALYSIS OF COVARIANCE 
(EFFECTS RELATIVE TO CONTROLS)* 

PPV . 
i vmite - Black 



Program 


Effect 




Effect 


t 




4 . 07 


3.62 


5.51 


2,98 




2.69 


2.37 


.2.44 


1.68 


Bank Streot^ 


s .52 


.37 


.270 


.24 


wregori 


1 1 2 


.87 


3.72. 


2.64 




2.25 


1.61 


1.90 


1.37 




.983 


.84 


-.840 


-.58 


Florida 


1 TO 


1 Oft 


0 ft 0 




EDC 


2.05 ' 


1.43 


1.27. 


a. 04 


Pittflburgh " 


4.42 


3.89 






REC 


.679 


.34 , 


3.89. 


2.08 


Enablers 


1.63 


1.19 


-.309 


-.23 


NPV 


2.37 


2.24 


, 1.69 


1.54 


Control 










Constant 
Fall PSI Coeff. 
Fall PPV Coeff. 
Fall Coeff. 

r2 

n 


20.9 ' 
.311 
.509 

111.00 
.609 

1011 


15.2 : 
/.408 

.558 ■^ 

120.60 
.564, 

• 1225 



* t > 1.96 is equivalent to p < .05 
t > 2,58 is equivalent to p < .01 
t > 3.27 is equivalent to p < .001 



- Table ^ Vi-il'' '. ■ ' '■'■■^ f \- r-'- 

213. ■ 

RESULTS OF ANAIiYSiS OF COVARIANCE 

t 

(EFFECTS RELATIVE TO CONTROLS)* 



WRTC 

White Black 



Program 


Effect 


t 


Effect 


t 


Par West 


1.74 


3.10 \ 


2,29 


3.35 




Ari^zona 


1.83 


,3.25 


2.49 


4.86 




Bank Street 


1.56 


2.29 


1.59 


3.54 






4.40 


4.40 


2.57. 


4.89 


Kansas 


3 . 59 


5.24 


4.45 


8.45 




Hlcih/Scope 


2.51 


4.33 


.764 


1.41 




Florida 


Z i J. 3 


^ Oft 




2.60 




EDC' 


1.15 


1.62 


3.36 


7.11 




Pittsburgh 


1.83 


3.24 








REC 


.636 


.71 


1.70 


2.49 




Enablers 


1.58 


2.72 


1.75 


3.41 




NPV 


1.87 


3.50 


1.53 


3.52 




. Control 












Constant 
Fall PSI Coeff. 
Fall PPV Coeff. 
Pail WRTC Coeff. 

n 


-2.12 
.201- 
.037 
.555 

67.52 
.497 

1 1033 


-1.82 

.108 
% .050 > 
.724 

96.08 
.512 

1295 



■ . ; I,' 

* t > 1.96 is equivalent to p < .05 
t > 2.58 is equivalent to p < .01 
t > 3.27 is equivalent to p < .001 
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■ ■ Ta^e ';■■ 
. - . ' • 214. 
RESULTS OF ANALYSIS OF COVARIANCE 

(EFFECTS RELATIVE TO CONTROLS)* 



WRTR 

White Black 





Effect 


t 


Effect 






1.90 


5.68 


1.92 


3.'ii;': 




2.26 


6,77 ♦ 


2.56 


6.08 




1.33 


3.28 


1.79 


4.86 


Oregon 


2 11 




1 .98 


4.59 


Kansas 


2.46 


6.06 


3.10 


7.17 


High/Scope 


1.74 


5.04 


1.16 


2.62 


Florida 


£ • JLD 




£ a OV 




EDC 


2.38 


5.67 


a. 77 


7.13 


Pittsburgh 


2.78 


8.25 






REC 


2.06 


3.87 


2.62 


4.70 


Enablers 


1.35 


'3.90 


1.64 


3.91 


NPV 


1.52 


4.80 


1.65 

\ 


4.64 


Control 










Constant 
Fall PSI Coeff. 
Fall PPV Coeff. 
Fa 1\; WRTR Coeff. 

^2 
n 


3.72 
.023 
.033 
.221 
34.18 
.333 
1043 


t 

2.89 

.069 _ . 

.023 
.275 
44.87 
.329 
1295 



* t > 1.96 is equivalent to p < .05 
t > 2.58 is equivalent to p < .01 
t > 3.27 is equivalent to p < .001 



RESULTS OF AmYSIS OF COVARIANCE 
(EFFECTS RELATIVE TO CONTROLS)*. 

WRTN 

■ I ■ ■ . ■ ■ . 

White- 



'215. 



Eilaok 





"E^'fecf"""'" 


' ■■■•r • 


Effect 


t 


i - 




1 1 •> 


^> fit 

* f vX 


3 53 








2 .47 ^ 


4.44 






1 00 
X i 


1 21 


2.49 V 


Oregon 


1 • 01 


CIA 




A 

H 9 OH 


Kansas 


4 • / /. 




^ « vo 






X • 00 






• 35 


Florida 


1.09 


1.54 


1.13 


2.12 


EDC 




6.78 


3.16 


6.17 


Pittsburgh 


1.47 


2.56 






REC 


1.47 


1.61 


1.62 


2.19 


Enablers 


1.07 


1.81 


,596 


1.08 


NPV 

• 


.864 


1.59 


1.57 


3.33 


Control 










Constant 
Fall PSI Coeff. 
'Fall PPV Coeff. 
P9II WRTN .'doeff . 

r2. 

. n - • 


-2.90 
.186 
.043 
.723 

87.57 
.561 

1043 


-2.86 , 

.181 

.042 

.764 
92.66 

.503 

1295 1 



* t > 1.96 is equivalent to p < iOS 
*t >'2.58 is equivalent to p < .01 
t > 3.27 is equivalent to p < .001 
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■ ^ RESULTS OF AKALYS IS OF COVARIANCE V 
€ (EFFECTS RELATIVE TO CONTROLS)* ■ 

- ' , . MRTD ' ■ ■ ■ 



• • ■ ^r:' ? .... ■ 
' White . ' Black 

t ' 









Effect 


■ t 






I * V ^ 6 


3 . 25 ? 


1.12 


3.98 




Arizona / 


• Off 


4 43 




4.95 




BAnK si;ireGu 


.342 


1.43 


.563 


3.14 




Oregon . 


1 •09 


coo 




10 81* 




Kansad . 


1 . 32 


5.4S 


1.96 


9.05 






■ .384 


1.89 


.345 


1.55 




Florida 


.341 


1. 39' 


• 0 /Q 


•J . «/9 




EDC 


i 1.20 


4.86 


1.16 


5.99 




Pittsburgh 


1.10 


5.52 








REC 


.525 


1.67 


.656 


2.35 




Enablere 


.385 


1.89 


.593 


2.82" 




1 

NPV 


.463' 


2.47 


.891 . 


4.99 




Control 












Constant- 
Pall PSI Coeff. 
Fall PPV Coeff. 
, B^allWRTD Coeff. 

n 


-1.05 
.085 
.023 
.302 

71.37 
.510 

1043 


' -1.03 
.079 
.016 
.464 
90.85 
.498 

1295 



* t > 1.96 is equivalent to p < .05 
t > 2.58 is equivalent to p < .01 
t > 3.27 is equivalent to p < .001 
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. ■ • 217. 

' results of analysis of covariance 
(effects relative to npv)* 

PSI 



White Black 







4. 






jrdr west 




1 42 


B9'0 

' ' 


1.13 


Arizona 


• 4 4 U 


• W 4 


071 


.15 




-1 .23 


-2.10 




-4.85 


Oregon 


1. Oft 




2-54 


5.28 




1.03 


1.76 


-.578 


-1.13 


High/Scope 


-.664 


-1.60 


-1.13 


-2.08 


Florida 


_i 1 ft 
-1.10 






17 


EDC 


1.24 


2.08 


.488 


1.28 


Pittsburgh 


■ .661 


1.68 






REC 


-1.87 


-2.05 


-.461 


- .58 


Enablers 


.089 


^ .21 


-1.38 


-2.90 


NPV 










Constant 
Pall PSI Coaff. 
Pall PPV Coeff • 
Pall Coeff • 

n 


7.68 
.531 
.122 

136.13 
;646 

985 


6.73 
.548 
.123 

148.69 
.597 

1217 



* t > A. 96 is equivalent to p < •05 
c > 2 •SB is equivalent to p < #01 
t > 3i27 is equivalent to p < .001 



Table VI-26 . , 

"218. . 

RESULTS OF ANALYSIS OF COVARIANCE * 
(EFFECTS RELATIVE TO NPV)* 



PPV 

White . Black 











Effect . 




t 


Par West 




1.69 


~ 1 

2 .29 


3.86 






2 « 44 


Arizcnd 




^ e A 

• 350 


« 47 


.591 






' « D J 


Bank Street: 






-.1 f A 


-1.40 






0 00 
* • * 4 


Oregon 






-♦24 


2.05 






1.99 


Kansas 








.238 






,23 


High/Scope 




-1.31 


-1.63 


-2.52 






•2.32 


Florida 




-.535 


-.47 


.349 






.41 


EDC 




-2.56 


-.22 


-.399 




• 


-.52 ; 


Pittsburgh 




2.00 


2.63 






REC 




-1.76 


-.98 


2.22 






1.38 


Enablers 




-.705 


-.B8 


-1.96 




-2.07 


NPV 
















Constant 
Pall PSI Coeff. 
Pall PPV Coeff* 
Pall Coeff. 

n 


23.60 
.296 
.507 

109.22 
.599 

I. 963 


16.90 
.414 
.554 

124.36 

.563 i 
1173 1 



* t > 1.96 is equivalent to p •< .05 
t > 2.58 is equivalent to p < .01 
t > 3.27 is equivalent to p < .001 
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Table VI -27 

RESULTS OF ANLAYS IS OF CQVARIANCB 
(EFFECTS RELATIVE Ta NPV)* 

WRTC 



White ' Black 



Program 


Effect 






■ t " 


Effect 






t 


Far West 


-.142 






-.41 


.753 






1.33 


Arizona 


-.058 






-.16 


.963 






2.88 


Bank Street 


-.337 






-.64 


.068 






.30 


Oregon 


2.48 






2.74 


1.04- 






2>96 


Kansas 


1.71 






3.22 


2.93 






6 .22 


High/Scope 


.615 






1.64 


-7.57 






•1.99 


Florida 


.257 






.47 


-.246 






-.82 


> 

EDC 


-.755 




-1.35 


1.84 






6.86 


Plt,i:8buroh 


-.054 






-.15 












-1.25 






-1.58* 








.31 


Enablers 


-.313 






-.83 


.217 






.65 


NPV 


















Constant 




.238 






.351 




Pall PSI Coeff. 




.202 






.108 




Pall PPV Coeff. 




.037 






.052 




Fall WRTC Coeff. 




.562 






.724 






66.16 




97.32 








.484 






.505 




n 


1003 








1254 









* t > 1.96 is equivalent to p < .0^ 
t > 2.58 is equivalent to p < .01 
t > 3,27, is equivalent to p < .001 



Table Vl-28 

. ' . 220. 

RESULTS OP ANALYSIS Of COVARIANCE ' » 

\ (EPPECTS RELATIVE TO NPV)* 

1 . ■ . ■ \' ■ ' . ' ■ , ■ ■; 

I • ■ , '' - ■ : " 



WRTR 



■r 




WKlte 




Black 




iprograitt 


• feffe'bt" 


t 


Effect . 


t 


j^ar West 


.399 




2.01 


.274 






.60 


l^rizona 


.752 




3.78 


.903 






3.35 


^ank Street 


-.202 




-.68 


.141 






.78 


i 

C>regon 


.609 




1.19 


.328 






1.16 


kaneas 


.945 






1.44 






5.03 


High/Scope 


.242 




1.14 


-.498 






1.62 


Florida 


.669 




2.17 


.740 






3.05 


Ebc 


, .881 




2.80 


1.13 






5.20 


Pittsburgh 


1.24 




6.07 . 










r 

REC 


.513 




1.14 


,.972 






2.14 


Ehablers 


-.145 




-.68 


: -.002 






-.01 


NPV 
















Cojistant 
Fall PSI Coeff. 
Pall PPV Coeff. 
Pail WRTR coeff, 

^2? 

r2 

'■n-i • ^ 


5.42 
.026 
.029 
. 210 
27.31 
.279 

1003 


4.59 
.071 
.022 
.268 
42.78 
.310 

1254 



* t > 1.96 is equivalent to p < .05 
% > 2.58 is equivalent to p < .01 
t > 3.27 is equivalent to p < .001 



Table VI-29 

221. 

RESULTS OF ANALYSIS OF COVARIANCE 
(EFFECTS RELATIVE TO NPV)* 

WRTN 

w 

White Black 





hitect 




• 

t 


Effect 


t 


Far Host 


-•138 




-•39 


1.04 


1»71 


Arizona 


2.34 




^•58 


• 901 


2 • 50 


Bank Street 


-•034 




-.06 


-.339 


-1 • 40 


Oregon 


• 107 




.12 


1.01 


2.68 


Kansas 


X • JO 








1 36 


High/Scope 


.781 




2.06' 


-1.35 


-3.28 


Florida 


.190 




.34 


-.428 


-1.32 


EDC 


3.97 




7.07 


1.60 


5.54 


Pittsburgh 


.599 




1.64 






REC 


.603 




.75 


.055 


.09 


Enablers 


.170 




\45 


-.969 


-2.70** 

< 


NPV 












Constant 
Pall PSI Coeff. 
Pall PPV Coeff. 
Pall WRTN Coeff. 

P. 
n 


-2.14 
.187 
.047 
.719 

88.44 
.556 

1003 


-1.38 

.181 . 

.045 % 

.756 
93.75 

.496 

1254 



' * r > 1.96 is equivalent to p < .05 
t > 2.58 is equivalent to p < .01 
t > 3,27 is equivalent to p < .001 
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Table VI -30 



RESULTS OF ANAIiYS^S OF COVARIRNCR 
(EFFECTS RELATIVE TO NPV)* 



222. 



WRTD 



White 



Black 



t&t West 
Arizona 
Bank Street 
Qjregon 
Kansas 
High/Scope 
Florida , 
EpC 

Pittsburgh 
REC 

Enablers 
WPV 



Constant 
Fall PSI Coeff . 
Fall PPV Coeff. 
Fall WRTD Coeff. 

R^ 

n 



.177 
;411 

-.125 
1.38 
.852 

-.083 

-.123 
.736 

- ,628 
.055 

-.081 



1.45 
3.35 
-.68 
4.38 
4.61 
-.64 
-.65 
3.81 
4.99 
.20 
-.61 



-.58 
.086 
.023 
.299 
69.78 
.497 

1003 



Effect 



t > 1.96 Lb equivalent to p < .05 
t > 2.58 is equivalent to p < .01 
t > 3.27 is equivalent to p < .001 



.228 
1 . 52 
-.323 
1.44- 
1.07 
-.544 
-.211 
.278 

-.235 
-.298 



.98 

1.11 
-3.53 
10.07 

7.33 
-3.49 
-1.71 



2.53 

-1.02 
-2.18 



-.157 
.079 
.017 
.452 
90.62 
.487 

1254 



Program 



liable VI-31 

RESULTS OF ANALYSIS OF COVARIANCE 
(EFFECTS RELATIVE TO MPV)* 

* 

White 



223. 



Far West 

Arizona 

Bank Street 

Oregon 

Kansas 

High/Scope 

t'lorlda 

EDC 

Pittsburgh 
REC 

Enablers 
NPV 



Black 



Effect 




t 


.Effect, 


t 


-.901- 




-1.09 


-.437 


-.30 




-.022 




-.02 


-1.86 


-1.81 




.028 




.02 


1.90 


r2.85 




4.91 




1.70 


-.833 


-.77 




-2.04 




-1.56 


' -.508 


-.44 




-2.64 




r2.83 


-1.71 


-1.50 




-1 . 54 




r-i.l2 


-.216 


-.25 




1.81 




1.35 


-.147 


-.19 




1.28 




1.48 








-4.06 




-2.23 


-3.45 


-1.94 




t1.61 




-1.86 


-3.61 


-4.23 

\ 




.052 
.476 
18.29 
.388 
418 ^ 


.061 
.406 
15.82 
.304 

476 


1 



Constant 
Fall PSI Coeff., 
Fall PPV Coeff. 
Fall ITPA Coeff. 

w- 
n 



* t > 1.96 is equivalent to p < .05 
t > 2.58 is equivalent to p < .01 
t > 3.27 is equivalent to p < .001 
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Table Vl-32 ; ' 

■ ' 224. 

RESULTS OP ANALYSIS OF COVARIANCR 

(EFFECTS RELATIVE TO NPV)* S 

ETS 



White • Black 



Program 


Y> JS ^ ^ ^ Al 

Effect 


• t 


Effogt 


t ' 


Far West 


.659 


1.97 


« A A 

1.20 


J. 74 


Arizona 


1.30 


1 W A 

3.78 


1.38 


3.39 


Bank Street 


OQ O 
• JOO 


.74 


A A 


. o / 


Oregon 


2.18 


A ^ M ■ 

2 .64 


A A J 

2 .34 


5.67 


Kansas 


1 . 0 4 


3 . 26 


2 . d2 


0 . 04 


High/Scope 


-.300 


-.83 


-1.14 


-2.44* 


Florida 


.733 


1.39 


.266 


.74 


EDC 


L.41 


2.67 


.647 


1.95 


Pittsburgh 


.735 


2.13r > 


^■^ ■ ■ ■ . 




REC 


-2.0$ 


-2.51 


-1.81 


:-2.65 


Enablers 


-.735 


-1.98 / 


.016 


;04 


NPV 




/ 
/ 






Constant 
Fall PSI Coeff. 
Pall PPV Coeff . 
Pall ETS Coeff. 

r2 

n 


4.11 
.180 
.045 
.404 
73.56 
.518 

974 


3.76 
.207 
.064 
.358 
75.39 
.451 
1207 



* t > 1.96 is equivalent to p < .05 ^«'» \ 

t > 2.58 is equivalent to p < .01 V\ 
t > 3.27 is equivalent to p < .001 ' ' 
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Preschool Inventory 

For Whites the largest effects were achieved by 
Oregon(5,94) , EDC (5.28) , and Kansas (^^10). The Oregon 
and EDC effects Were significantly above NPV. Smallest 
efj^cts were for REC (2*27) and Bank Street (2.90), These 
were significantly below NPV. For Blacks, Oregon (5.86) 
and Pdr West (4.24) had the largest effects* Only Oregon 
was significantly above NPV. Smallest effects were for 
Bank Street (1.80) r Enablers (.197), and High/Scope (2.25) 
All three were significantly below NPV. For both Blacks 
and Whites, all models performed significantly better 
than the Controls. 

Peabody Picture Vocabulary Test 

For Whites/ Pittsburgh (4.42) and Far West (4.07) 
ha^ the largest effects, both significantly above NPV. 
Smallest effects were for Bank Street (.52), REC (.68), 
and High/Scope (.98) . None of these isrere significantly 
below NPV. For Blacks, Far West (5.51), REC (3.89)^ and 
Oregon (3.72) had the largest effects, with Far West and 
Oregon significantly above NPV. High/Scope (-.84), 
BjQ^blers (-.31) and Bank Street (. 27) were lowest, all 
three significantly uelow NPV. While all mode^ effects 
for Whites and 8 of 10 for Blacks were positive, most, 
effects were not significant. 
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WRAT Copying Marks 

Largest effects for Whites were achieved by Kansas 
(5.24), Oregon (4.40), and High/Scope (4.33). Kansas and 
Oregon were significantly above NPV. REC (.71), EDC (1.62), 
and Bank Street (2.29) Were loW| although none were 
significantly below NPV. For Blacks, Kansas (4.45) and 
EDC (3.36) were high and significantly above NPV, while 
High/Scope (.76) was low and significantly below NPV. For 
Whites 9 of 11 model effects were significant and for 
Blacks 9 of 10. 

WRAT Recognizing Letters < 

For Whites, Pittsburgh (2.78), Kansas (2.46), and EDC 
(2.38) were high, all significantly above NPV. Enablers 
(1.27) 'and Bank Street (1.33) were low, but not Significantly 
below NPV. For Blacks, Kansas (3.10), EDC (2.77), REC 
(2.i62), and Arizona (2.56) were high, all significantly 
above NPV. High/Scope (1.16) was lowy bgt not significantly 
below NPV. All model effects were significant. 

WRAT Ncuning Letters 

For Whites, EDC (4.87) and Arizona (3.23) were high, 
both significantly above NPV. Bank Street ,(. 84) was low, 
but not significantly below NPV. For Blacks, EDC (3.16) 
Far West (2.61), Oregon (2.53), and Arizona (2.47) were 
high, with all except Far West significantly above NPV. 
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For Whites 5 of 11 model effects were significant, and for 
Blacks 8* of 10* 

WHAT Reading Numbers 

For Whites, Oregon (1.85) and Kansas (1.32) were high, 
both significantly above NPV. Bank Street (.34), Florida 
(.34), High/Scope (.38), and Enablers (.39) were low, 
although none was significantly below NPV. For Blacks, 
Oregon (2.33) and Kansas (1.96) were high, and both were 
significantly above NPV, while High/Scope (.35) and Bank 
Street (.56) were low, and significantly below NPV. For 
Whites 6 of 11 model effects were significant and for 
Blacks 9 of 10. 

ITPA Verbal Expression 

^ince the Control children did not take the ITPA, 
no comparisons with them were possible. From the comparisons 
with NPV, however, we find for Whites that Oregon (4.91) 
vashighest, though not significantly above NPV. REC (-4.06) 
and High/Scope (-2.64) were lowest, both significantly below ^ 
NPV. For Blacks we find that all had smaller estimated 
effects than NPV, with the Enablers (-3.61) significantly 
lower. 

ETS Enumeration 

As for the ITPA, no comparisons with Controls were 
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possible. From the comparisons with NPVi for the Whites 
Oregon (2*18) and Kansas (1.64) i;,;ere high/ although neither 
was significantly above NPV. REC (~2.06) and Enablers 
(-.74) were both significantly below NPV. For Blacks, 
Oregon (2.34) and Kansas (2 . 22) were high and both signifi-> 
cantly above NPV, while REC (-1. 81) was significantly below 
NPV. 

Summary of ANCOVA Result? 

A3 for the previous analysesjp we will present in this 
section evidence furnished by the ANCOVA beiaring on our 
thxee major questions. 

1. To what extend does a Head Start experience 

accelerate the rate at which disadvantaged pre- 

f7 - . /, ' . 
schoolers acquire cognitive skills? 

* . ■ - ■' . 

« . » ■ ■ ■ 

Our evidence here come from the comparisons between 
the Control and Head Start children* For three of the six 
tests taken by the Controls (PSI, WRTCr WRTR) / nearly all 
the PV and the NPV children do significantly better 
than the Controls for both Blacks and Whites. For two 
tests (WRTN/ WRTD) fliost of the models perform better. Only 
for the PPV do the Control and Head Start children perform 
comparably. . 
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2> Are the Planned Variation models, simply by virtue 
.of sponsorship, more effective than ordinary 
non-sponsored Head Start programs? 

On the whole it appears that PV and NPV programs' are 
similar in effectiveness. As a very tough measure of 
over-all NPV performance, we observe that fox White children, 
of a total of* 88 model effects on the 8 tests, 51 were 
. above that^of NPV and 37 were below. For Blacks, of 80 
model effects, 46 were above and 34 below. 

3. Are some PV models particularly effective at 
imparting ceiftain skills? 

Table VI-33 presents a summary of inter-model , compari- 
sons; In declciring a model particularly effective or 
ineffective for a given test, we have considered the size 
of the estimated difference in 'effects between the model 
and the Controls, the significance of the difference 
between the model' and NPV, and the consistency a^oss 
racial groups* The effects noted in Table VI-33 seem 
fairly evenly spread across the 8 tests. It is interesting 
that no model has both positive and ijegative effects. 
Oregon and Kansas are overall most impressive, each with 
5 positive effects out of 8 tests. 
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; Table VI-33 

Summary of Relative Model Effectiveness 
' Based on Analysis of Covariance* 



++ Indicates model appears to be highly effective. 
+ Indicates evidence for above average effectiveness. 
Indicates evidence for below average effectiveness* 
— Indicates model appears to be highly ineffective. -^^^ 



Model 



1 PSI 



PPV 



WRTC 



WRTR 



WRTN 



WRTD 



I TP A 



ETS 



Par Weist 

Arizona 

Bank iStreet 

Oregon > 

Kansas 

High/Scope 

Florida 

EDC ' . 

Pittsburgh 

Enablers 



++ 



+ 



++ 

+4- 



+4- 
++ 



*REC not included because with only one site we felt it 
unfair to draw any conclusions.. ." ^ 
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Chapter VII 
Resistant Analysis 

I" 

Introduction and Theory 

It was clear "to us that the pre-test was the most 
important variable to control in making post*-test com- 
parisonSv We thought it would be "worthwhile to do some 
exploratory analysis to determine the relationship between 
fall and spring test scores, broken down by program and 
possibly background characteristics. All our previous 
analyses used ineans as summary measures of distributions 
of effects, and relationships were fitted via ordinary 
loast-*squares techniques. While we have confidence in these 
analyses, we felt it ,\yould be nice to have at leas>: one 
analysis using other summary statistics and fitting methods 
which would be particularly robust , or resistant to de- 
partures from the usual assumptions underlying the standard 
procedures. We were thus led naturally to the recent work 
of John Tukey (1970)^ Tukey's exploratory data analysis 
techniques enable the analyst to comb a set of data for use- 
ful information without unwieldy computations and formal tests 
based on stringent assumptions. We found the resistant 
fitting technique particularly appropriate. With it, we 
could conveniently fit a model of the f orm * , 
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where Y^^^ and Y^^ are pre- and poet-test scores for 
individuals 1 ir^ group j, ej^j is the error termr and f ^ 
is a transformation or re-expression of Y^^ consisting 
of any power or the logarithm. For example r on the PSI/ for 
White children with no prior preschool in Far West, we 
fit the model: 

y« « -.32 + 19.13 logY 

For Black children with no prior preschool in Far West, we 
obtained • 

Y' ^ 26.73 - 80.89 



Note that the class of models described by (7.1^ 
can be characterized as linear in terms of the re-expressed 
pre-test score. The details of how the appropriate re- 
expression is se^cted, and the slope and intercept 
estimated can be found in Appendix written by Sharon 
Hauck. In ordinary least-squares fitting, outlying 
observations (those with very large values of ©j^j) exercise 
a strong influence in determining the fitted curve. 
Resistant fitting is much less sensitive to such outliers. 
Thus/ it provides a fairer representation of the data 
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in a situation where nearly all observations reflect a 
systematic relationship^ but there are a J«?w "wild" 
observations. Since jbhese wild observations are not given 
much weight in the curve-fitting, they will also stand out 
more strongly than in a least-squares regression when we 
look at the residuals 

In trying to apply the resistant fitting technique to 
the various tests in our battery, we found thv\t the floor 
and ceiling effects of the WRAT subtests made it virtually 
Impossible to implement the fitting algorithm. We decided 
to take a different tack with three of the WRAT aubt^sts. 
For the WRAT Recognizing Letters, Naming Letters, and 
Reading Numl^ers, it seemed that many children were achieving 
an effective maximum, so that their potentliil gain was 
strongly dependent on where they started out. It seemed 
reasonable to consider these as criterion-referenced tests. 
We, therefore, set a level for each test which we ^elt 
corresponded to reasonable mastery of the subject matter. 
^ For each child we could then note simply whether or not 
he reached this criterion. Looking at all children with a 
given pre-test score (or narrow range of scores) we could 
than see for each program the proportion reaching criterion. 



♦Residual = Observed Value - Pittei Value 
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It would, of course, be desirable to control for other 
variables as well. Since ethnicity seemed to have a strong 
relationship to outcomes, we considered Blacks and Wliites 
separately,* Sample sizes were not adequate for further 
breakdown by other background characteristics. 

The other five tests did not seem to us suitable for 
the criterion-reference analysis, since the, did not 
involve such clear-cut, concrete skills and it was not 
clear how to set a criterion for subject mastery. For 
each of thesQ, we performed a resistant fitting analysis. 
We broke the children in each program out according to 
ethnicity and whether or not they had any prior preschool 
experience. For' each sub-class In which there were at 
least 20 children, we "then fit a model of the form de- 
scribed by equation (7.1). We studied the resulting 
functions/ but no strong patterns became obvious. We de- 
elded to attempt to obtain simple comparisons among programs 
by developing a resistant analog to the usual least- 

squares analysis of covarlance. 

«> . 

Suppose that for any ehtnlclty by prior preschool 
experience sub-class ^ we can rep^reoent the relationship 
between fall and spring tests by a mo^el of the form: 

^Ij' " ^ ^^^^ij^ ^ ®ij ^'-2) 



*There w6re not enough Spanish-Americans to make the analysis 
for them worthwhile. 

ERJC . 
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That l8, W6 assume that the re-expression f and the slope 
B are the* same for all programs. Thus, as In the ANCOVA/ 
beccxnes a measure of relative program effect. 

From the various re-expressions found in the model- 
fitting described above, we selected a compromise re*- 
expression reasonably acceptable for all programs i though 
perhaps not optimal for any particular program. For each 
progreuo, a model was then fit using the common re**expression. 
This resulted in a set of up to 13 slopes,* one for each 
program. From this set we determined a compromise slope, ^ 
hopefully reasonable for all programs. Having decided on 
both 3 and f, we could now estimate by taking a kind of . 
weighted average of the deviations of the Y^^'s in group 
j from th" function 0f(y^^). The details of the steps 
described above can be found in Appendix E. 

To calculate a progreun "effect" we took the median of 
the estimated Qj's and subtracted this from each of the 
ttj's individually. The result is analogous to that of a 
standard one-way ANCOVA with effects competed around a 
grand mean of Vail pr6gr£ims« To see how our results com- 
pared with those of the more traditional approach, we 
carried out such an ANCOVA. 

^J^ , ^ , . " 

. .f* ■ 

*The actual number was the number of programs with at ^least 
20 children. 
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Results of the Criterion-Reference Analysis 

Results of the criterion-reference analysis for the 
WRTR, WRTN, and WRTD appear in Tables VII-1 through VII-5. 
For the WRTR the maximum score was 10, and we decided that 
t6 reach criterion a child must achieve a score of at least 
9. For the WRTN our criterion was 10 out of 13 correct. 
The WRTD requires the child to read the num^^rs "3, 5, 6, 
17, 41," It seems that 17 proved quite difficult and 41 
much too difficult for our sample. We/ therefore, decided 
that 3 of 5 correct seemed a reasonable criterion. 

For Blacks and Whites separately, we looked at all 
children with fall scores in certain narrow ranges, and 
recorded the number reaching criterion and the number 
failing to do so. We also calculated the proportions of 
PV children, NPV children, and Control children reaching 
criterion. Following are summaries of the results. Note 
^hat we elected not to perform significance tests for 
differences between proportions. There wore so many 
possible inter-dependent tests that significance levels 
would be severely compromised. The reader has, of course, 
from Tables VII-1 through VII-S all the information necessary 
to carry out any tests he may deem useful. 



Table Vll-1 

RESULTS OF CRITERION REFERENCE ANALYSIS FOR 

VmTR 
Black 
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Table Vll-2 

RESULTS OF CRITERION REFERENCE ANALYSIS FOR 

* WRTR 
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resi;lts of criterion referencs analysis for 
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ftBSULTS OF CRITERION BEFERENCE ANALYSIS FOR 

WRTD 
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Table Vll-5 

RESULTS OF CRITERION REFERENCE ANALYSIS FOR 

WRTD ' . 

White 



241. 



Jail score = 0 



+» 

(A 

$ 



4J 
CO 



g 

o 



(0 

(0 



0 
H 



t 



■M 



> 

H 
O 



27 


29 


5 


3 


17 28 


5 


14 


37 


5 


22 51 


3 


192 


54 


44 


23 


0 


12 38 


10 


8 


49 




39 104 


25 


2'86 




















* 6 7 




t ^ W M 










Fall 


Score 




1 










10 


15 


2 


2 


4 2 


2 


3 


12 


0 


8 8 


1 

•Ik 




8 


4 


1 


0 


0 3 


1 


0 


4 


0 


5 7 


1" 


26 




















.533 


;5oo 












Pall 


Score 




2 










6 


n 


2 


1 


3 8 


1 


1 


-4- 


1 


5 6 


2 , 


44 


2 


1 


1 


0 


0 2 


0 . 


0 


0 


1 




1 


7 




















.75C) 


.667 


.861 










Fall 


Score 




3 




/ 

- ■ I 






16 


21 


4 


3 


4 17 


7 


10 


6 


0 


3 15 


1 


91 


2 


0 


0 


0 


0 0 


0 


0 


0 


0 


0 0 


0 


2 




















1.00 


1.00 


.978 



Spring 3 
Spring 3 



60 Spring ^ 3 
26 Spring 3 



44 Spring 3 



Spring 3 



91 Spring 3 
Spring 3 



ERIC 



242. 



WHAT Recognizing Letters 

For Whites, we noticed that the proportion of PV 
children reaching criterion was higher than the proportion 
of NPV children for all 5 fall test score levels. It. 
appears that Kansas, Pittsburgh, and possibly Arizona and 
EDC are particularly effective. For Blacks PV children 
also did consistently better than NPV children except for 
those reaching criterion in the fall. Kansas seems par- 
ticularly effective, and possibly Arizona and EDC. Note 
that for most programs and fall scores, the proportion 
^reaching criterion is smaller for Blacks than for Whites* 

WRAT Naming Letters 

It turned out that nearly all children scored 3 or 
less in the fall, and very few of these reached criterion 
in the spring. Thus, it was difficult to make puch of the 
results. • There is some evidence that Arizona, Oregon, and 
EDC may be particularly effective. 

WRAT Reading Numbers 

For both. Whites and Blacks the majority of children 
scored 0 in the fall. There are enough with other scores 
to be worth presenting, but too few to draw conclusions. 
For those with fall scores of 0, the proportion reaching 
criterion was higher for PV than for NPV for both Blacks and 
Whites. The proportion for Whites was higher than that for 

ERLC 
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Blacks for both PV and NPV children. For Wh»ites, Kansas 
and possibly Oregon seen\ particularly effective. For 
Blacks, Oregon is outstanding and Kansas also particularly 
effective. ^ ... 

Results of the Resist a nt Analysis of Covariance 

As explixined above, we initially attempted a re- 
sistant fit to describe the relationship between falX and 
spring scores for each of our tests. As a result of 
ceiling and flopr effects, this proved particularly diffi- 
cult for the WRAT subtests^ For the other tests, it was 
difficult to summarize the results meaningfully, and we 
decided to attempt the resistant ANCOVA analog. Unfor- 
,tunately, for the ITPA and ET3, many of the models did not 
contain enough children to justify their inclusion, and 
the choice of a compromise re-expression to be applied to 
all programs was very difficult. We, therefore, decided 
to carry out the resistant ANCOVA for the P&I and PPV only. 

The results appear in Tables VII-6 and VII-7. For 
each ethnicity by prior preschool sub-^classes, we have . 
calculated an estimated effect, for each program containing 
at least 20 children, relative to the median effect for all 
such prograuns. As a comparison, we have also ^performed 
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^ ^ RESULTS OF RESISTANT. ANALYSIS OF COVARIANCE FOR 

PPV 
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RESULTS OF ANALYSIS OF COVARIANCE FOR 

PSI 
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RESULTS OF ANALYSIS OF COVARIANCE FOR 







P£V 
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an ordinaty lea^t-squares ANCOVA with effects computed 
about *t he grand pean.* These results appear in Tables 
VII-8 and VIIr-9.| We now present brief summaries of the 
resistant ANCOVA \re suits. 

Preschool jj;nvei]tto]^y 

For White cl^ildren with no prior pre-school, the 
Controls have bi^ far the lowest effect (-4.1), For 
Blacks with n6 prior pjreschool. Far West (3.1) and Oregon " 
' (3.0) are hi^^h, and for Spanish Americans, Oregon (3.0) \ 
is outstanding. The effects for both Whites and Blacks 
with prior preschool are rather homogeneous. Oveijrall, it 

appears that Far West and Oregon are particularly effective 

\ 

Bank Street and\ Control particularly Ineffective. The 



results for the Standard ANCOVA seem remarkably consistent 
with those of our\ resistant ANCOVA. 

r 

' Peabody Picture Vodabulary Test 

, For White children with no prior preschool experience, 
program effects seem <^uite homogeneous. For Blacks with no 

v i 

*Theife wete three main differences between these ANCOVA' s 
r^^and those carried^ out in Chapter VI, First, the children 
were broken down by prior preschool experience as well as 
ethnicity. Second, the effects were ccxnputed about the 
grand mean (an unweighted mean of the spring means for all 
programs) rather than relative to the Controls. Third, the 
analysis was carried out using an unweighted means approach 
rather than exagt least-squares « 
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prior preschooli Oregon (4*1) and perhaps Florida (2.3) 
seem particularly effective. For Spanish-Ameiicans, NPV 
(4,1) does best. For Whites with preschool, Pittsburgh 
(6.4) seems highly effective, and Arizona (-2.6) possibly 
ineffective. For Blacks, program effects are quite homo- 
geneous. The general profile of effects from the standard 
ANCOVA is similar, although the magnitudes of effects 
differ, 

Summary 

We have results for only five tests (PSI, PPV, WRTR, 
WRTN, WRTD) from the analyses discussed in this chapter. 
As in previous chapters, we present here the evidence pro- 
vided by these analyses bearing on our three major questions. 

1. To, what extent does a Head Start experience 
accelerate the rate at which disadvantaged pre- 
schoolers acquire cognitive skills? 

For all tests except the PPV, the Controls appear to 
do substantially worse than both the PV and NPV children. 
On the PPV, Head Start and Control results are comparable. 

2. ' Arq the Planned Variation models, simply by virtue" 

of sponsorship, more effective than ordinary non- 
sponsored Head Start programs? 
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For each of the 5 testo except the PPV, the'PV 
programs as a whole perform slightly better than NPV. 
For tbo PPV, their performance is roughly equivalent. We 
are inclined to attribute the slight superiority of PV 
to a couple of particularly effective models, so that, 
except for these, the effects of PV and NPV are comparable. 

3. Are some PV models particularly effective at 
imparting certain skills? 

Table VII-8 presents a summary of inter-model compari- 
sons. In deciding whether to declare a mbdel particularly 
effective, we have considered whether*the proportion 
reaching criterion is consistently higher than the over- 
all PV proportion for all ethnic groups and fall scores. 
For the PS I and PPV, we^ have considered the size of 
effects and their consistency over the ethnicity by prior 
preschool sub-classes. Note t^at out of 11 positive 
effects, 7 are for the acaciemic models*^^ (Oregon, Kansas, 
Pittsburgh). Overall, Oregon is most impressive. 
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Table VlI-10 

SUMMARY OF RELATIVE MODEL EFFECTIVENESS 
BASED ON RESISTANT ANALYSIS* 



++ Indicates model appears to be highly effective, 

+ Indicates evidence for above everage effectiveness. 

- Indicates evidence for below avierage effectiveness. 

— Indicates model appears to be highly ineffective. 



Model 


PSI 


PPV 


WRTR 


WRTN 


WRTD 


Par West 


■ > 
+ 


i 








Arizona 






+ 


+ 




Bank Street 












Oregon^ 


++ 






+ 


++ 

* 


Kansas ; ^ 






+ 




+ 


High/Scope 












Florida 












EDC 








+ 




Pittsburgh 




+ 


+ / 




* 


Bnablers 










s» . 


NPV 













*REC not included because with only one site we felt it 
unfair to draw any conclusions. - 
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Chapter VIII 

Background Character iatic by Program Interactions 

Introduction 

The previous four chapters have attempted to 
present a picture of the pattern of overall effects of 
various programs* In this chapter we explore the question 
of whether the relative effectiveness of various programs 
is related to Certain child background characteristics* 
Featherstone (1973) studied this question using the 1969- 
70 and 1970-71 data. We are in no way trying to replicate 
her careful and thorough study. Without a carefully 
designed randomized experiment, the problems involved 
in estimating interaction effects are much more difficult 
than the already difficult problems involved in measuring 
main effects (see Chapter III). Definitive conclusions 
from our data are virtually impossible. Nonetheless, we 
felt that a modest effort to see what interactions are 
suggested by the data, and how they relate to Feather- 
stone's general conclusions, would be valuable. 

The 6utcomes Featherstone used were the Stanford- 
Binet IQ and the 64-item PSI* Since neither of these tests 
was given in 1971-72, comparisons with her results are 
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difficult/ The background characteristics she considered 
were initial (fall) IQ, prior preschool experience, sex, 
age, socio-economic status^ ethnicity (Black and White 
only) and cognitive style (as measured by the Hertzig- 
Birch coding of the Stanford-Binet) . We have no measure of 
IQ for the 1971-72 cohort, and, although a version of the 

Hertzig-Birch scoring system was used with the 32-item 

m 

PSI, we felt the system was too experimental to use at 
this time. As for SES^ we felt that from the standpoint 
of reliability and impact on test scores, mother *s edu- 
cation was our best variable. We therefore decided to. 
look only at sex, mother's education, ethnicity (Black 
and White only), age, and prior preschool experience. 

Since the interpretation of interaction effects is 
sometimes confusing, it may be useful to explain exactly 
what they mean in this context, and why they are so diffi- 
cult to estimate. For bimplicity, suppose we have two 
programs, A and B, and that sex is the background variable 
of interest. Assume we have some measure of program 
effectiveness (e.g., residual, adjusted mean) ad that 
in terms of this measure we obtain the hypothetical results 
displayed in Figure Vlll-la. In this case there is no 
interaction between sex and program, since the difference 
between the effects of the two programs is 4 for both boys 
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figure VIII-1 



ILLUSTRATION OP PROGRAM BY SEX INTERACTION 



Male 



Female 



Male 



Female 



Male 



Female 
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Program 
A B 



Interaction 



12 


8 


14 


10 


1 

(a) 


12 


8 


14 


12 


(b) 


12 


8 


10 


12 



(c) 



(12-8)-(14-10) » 0 



(12-8}-(14-12) = +2 



(12-8) -(10-12) = +6 
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t , " 

and girls. In situation (b) ^ on the other hand^ the 
, difference between program A and B is larger for boys than 
for girls. Thus, sthe relative effectiveness of the pro- 
graireis related to the child's sex. We have a program- 
by-sex interaction. Finally, in (c) we have a disordinal 
interaction. Not only is the difference in effects greater 
for boys, but the direction of relative effectiveness of 
the two programs is actually reversed. Program A is better 
than B for boys, while program B is better for girls. 

Notice that an interaction is rfealiy#a difference ^ 
of differences. Thus, in estiioating an interaction effect 
from a finite scimple, a small sample in any of the four 
cells can lead to imprecise estimates (i.e., large variance). 
In the extreme, an empty cqII makes the estimation im- 
possible. If, for example, there were no boys in program A, 
no statistical procedure could provide a reasonable 
estimate of the interaction, 

Mea:hodology 

We decided that the simplest way to measure inter- 
action effects would be to use the "combined" residuals 
derived in Chapter V as an outcome measure* Recall that 
the residual is an estimate of the effect of the program 
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in which the child is enrolled over and above what we 
would expect on the basis of natural maturation, * We <I:an 
perform a two-way analysis of variance, with program 
and a background variable as factors • This ANOVA will 
provide an F-test for the significance of the overall 
interaction effect. Looking at the pattern of cell means, 
we will hopefully be able to int4ilpret any interactions 
detected. As an added benefit, we will also obtain P- 
tests for the main effects of program and background 
characteristics,. If we observe a large main effect corres- 
ponding to a background variable which is unevenly dis- 
tributed across the various programed, there may be some 
bias in the magnitude of the estimated model mean residuals. 

Because the design is quite unbalanced (unequal cell 
sizes) an exact least-squares solution would be quite 
complex • We therefore carried out an unweighted means 
analysis. Unfortunately, for some background variables, 
the design may be so unbalanced thc4t the F-test resulting 
from the unweighted means analysis may be misleading. 
Since our primary interest is in the estimation of effects 
rather than formally testing hypotheses, this does not 
concern us overly. Moreover, in carrying out ANOVA 's on 
the six tests for which we have computed residuals, for 
each of the five background characteristics, we perform a 
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large number of statistical tests • Thus the formal signifi- 
cance level of any individual test might be compromised 
even with an exact least-squares analysis. 

Results of Interaction Analysis 

In this section we present the results of the 
interaction study. Detailed results are presented in 
Tables VIII-1 through VIII-28. We first present brief 
summaries for each background characteristic, followed by 
some concluding comiaents. 

Sex . The only significant main effects for sex occur on 
. the PSI and PPV. There are small differences favoring 
boys on both. There are no significant program-by--sex 
interaction effects on any of the tests. The' overall 
pattern of relative model effectiveness is quite similar 
for boys and girls. 

Ethnicity . All tests except the WRTD show significant 
main effects for ethnicity. The PSI and WRTC effects 
favor Whites, while the PPV, ITPA, and ETS effects favor 
Blacks. Only the WRTC and WRTD have significant inter-- 
action effects. The WRTC effect (p<.001) is largely 
attributable to High/Scope and .EDC. High/Scope was highly 
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effective for Blacks and below average for Whites • These 
results may well be attributable* to site characteristics 
other than ethnicity, as ethnicity and site are confounded. 
Fran Table II-2, we see that of High/Scope*s two sites 
Fort Walton Beach was 75.3% Black, while Central Ozarks 
was 100% White. In EDC virtually all the White children 
were in one of the two sites. High/Scope jmay also be .V 
responsible for the WRTD interaction effect (p«.05). 

Age . We divided the age range into three categories: 
under 54 months, 64 to 60, over 60. There were only four 
tests for which the age distrubution was sufficiently 
balanced to allow us to carry out the analysis. Even for 
these four (PSI, PPV> WRTC,. WRTD), it was necessary to 
eliminate the Oregon model, since it contained no children 
under 54 months of age. All tests except the PSI have, 
significant main effects for age, and the PSI , effect 
is almost significant (p«.08). For the PSI and PPV, 
age is negatively related to residual size (yoUnger children 
gain more) , while for the WRTC and WRTD it is positively 
related. Interaction effects were significant ^or the PSI 
(p«.003), WRTC (p < .001) , and WRTD (p^.04). The pattern 
of interaction effects is difficult to interpret.- Moreover, 
several models contain very few children over 60 months of 
age. If we look only at those' children 60 months old 

-J ^ * 
, <^ * 

\ ^ 
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or younger ^ the pattern of relative program effectlv'eness 
apj^eats fairly oorie^istent across the two other age groups. 

• - 

Prior Preschool Experience , Three of the six tests show 
significant main effects favoriag children with no prior 
preschool experience. These are the PSI (p < .001), 
PPV {p<,.001). and'WRTC (p».q25)* Recall, however, that in' 
Chapter V we noted that the residuals may be less valid 
measures of program effectivenss for children with prior 
preschool experience than for those without* Thus an appar- 
ent prior preschool effect might really be an artifact of 
the way in which the residuals were coniputed. Significant 
interaction effects occur on the PSI (p ^ .001) and WRTD 
(ps^.Ol). The PSI interaction may well be at least in part 
a spurious artifact of the unbalanced design. The Kansas 
model in particular appears to do terribly for children 
with prior preschool^ experience, but the mean for this 
cell ^ is based on a sample of only seven children. Thus, 
although the data suggest the possibility that relative 
model effectiveness on the PSI is related to prior preschool 

experience, we cannot interpret this interaction with much 

♦ 

confidence. Note that REC and Enablers are both more 
effective than average for children with prior preschool 
and less effective for those without • 
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, Mother's Education 

There are significant main effects for mother's 
education on the PSI (p < .001), PPV (p<,001), WRTC 
(p«.01), WRTD (p^.Ol), and ETS (pa.003). The PSI, PPV, 
and vmTC effects reflect a negative relationship between 
mother's education and residual size. There are no sig- 
nificant interactiorf effects on any of the tests. 

Featherstone (1972) found generally that relative 
model effectiveness tended to be related to variables 
which describe the child at a particular stage of develop- 
ment rather than to permanent, unalterable characteristics* 
Our result^ generally corroborate this finding. The fixed 
characteristics we studied (sex, ethnicity, and mother's 
/education) showed very few significant interaction effects. 
Age and preschool experience, on the other hand, ^ yielded 
a fair number. These effects were not, however, easy to 
/ interpret and the unbalanced design severely limited our 
confidence in their validity. 
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RESULTS 6g INTERACTION ANALYSIS ' 

' ESI 4 ' 

9 









(' 


ROW Mar^lTlaXS, 




> 


2 » 63 




^ 2.71 










1.60 






— •JO 




-.22 


u^egon 








1.03 


V A n A a 




^ 00 

it % itC 




2 . 55 


HX^n/bCOpd 




« 73 




1.08 


Florida 


1.09 


1.99 


t 


1.54 V 


BDC 


2:24 


.162 




1.93 


Pittsburgh 


3.28 ' 


1,62 


/ 


2.45 


REC 


2.59 


2.36 




2.47 


Enablers 


2.36 


1.40 


• 


1.08 


Control 


.63 


-.33 




.25 


NPV 


1.80 


.94 


■ * 


1.37 


Column 


2.05 








Marginals 


1.43 




" '1^74--' 




P 








Program 


8,433 


<,001 




• 


Sex 


11.438 


^.001 






Program X Sex 


, 1.040 

- / 


.41 


• 


* 



*Marginal8 are unweighted averages of cell means. 
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RESULTS OF IMTERACyiON ANALYSIS 
PPV 



Program 


Male 


Female 


/ ROW Marginals* 


Far West 


5 . 28 


6.41 


■ r - 


5.84 


Arizona 


5.42 


S^.OO 




5.21 


Banjc^Streot 


5.41 


\- 5.64- 




5.53 


Oregon 


8.16 


6.79 




7.47 


Kansas 


7.15 


5.75 




6.45 


Hlgh/Soot>e 


3.99 


3.09 




3.54 


Florida 


6.96 


4.54 




5.75 


EDC 


5.75 


6.49 




6.12 


Pittsbvirgh 


4.96 


6.92 




5.94 


REC 


12.09 


7.89 




9 . 99 


Bnablers 


4.38 


4.16 




4.27 


Control 


7.22 


4.92 




' 6.07 


NPV 


6.87, 


5.83 




6.35 


Column 
Marginals 


6.44 


5.65 , 


^ 


6.35 




F 


Si cinificancG 


> 




Program 


4.799 


<.001 








.4.052 


!045 






Program X Sex 


1.365 


.176 







r 



.'♦Marginals are unweighted aver.ages of cell means. 
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RESULTS OF INTERACTION ANALYSIS 
WRTC 



Proqram ^ 


V Male 


Female 




ROW Marginals* 


Par West 


1.40 


1.65 




1.52 


Arizona 


2.15 


2.18 




2.17 


Bank Street 


1.35 


1.42 




1.39 


Ojregon 


■ 3.86 


2.79 




3.33 


Kansas 


. 4.01 


4.80 




4.40 , i 


High/Scope 


2.70 


1.88 




2.29 , 


Florida 


1.36 


2.55 




1.96 


EDC 


2.40 


2.97 




2.67 


Pittsburgh 


1.61 


1.24 


1 


1.42 


REC 


1.32 


1.34 j.' 




i. • j 


Enablcrs 


1.74 


2.13 




1 0 


i^OTiuxrox 


.84 


.05 




.45 


Kir V 


1.83 

j 


2.05 




1.94 


Column 
Marginals 


2.04 


2.08 




2.06 




F 


Sfqnif IcancG 




i 


Program 


13.03 


<.001 






Sex 


.06 


>.5 






» 

Program X Sex 


1.43 


.15 







*Marginal8 are unweighted averages of cell means. 
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RESULTS OF JNTERACTION ANALYSIS 





* 


WRTD 




« 


Program 


Male 


Female 




* 

(ROW Marginals^ 


Far West 


.94 


.86 




.90 


Arizona 


1.08 , 


1.25 




1.16 


Bank Street 


.42 


.57 


/ . 


.50 


Oregon 


2.71 


2.63 




2.67 


Kansas 


1.77 


2.03 




1.90 


High/Scope 


.74 


.68 






Florida 


.32 


l!o5 




.69 


EDC 


1.17 


1.35 ; 




1.26 


Pittsburgh 
REC 


1.28 
.59 


.86 
.82 




1.07 
• .70' 


Enablers 


.71 - 


i 

.71 




.71 


Control' 


.30 


.05 




.18 


NPV 


,84 


.88 / 




.86 


Qoluron 
n^gmai s 


.99 


1.06 


,1 ■ < ■ ■ .1 


1.02 




F 








Program ^ 


37.54 


< Oft 1 






Sex 


1 . 26 


.26 






Program X Sex 


1.72 


.06 

* * 






♦Marginals are 


unweighted averages of coll 


means. 
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, RESULTS OF INTERACTION ANALYSIS 

1 ' ^ J " I ' . , I i I I l l 11.1 , I , „ ,l I ■ ,>l . I N I, 



* 

l' 




4 

ITPA 




Program 


■ Male ' 


Female j ' 


« 

ROW Marginals* 


Far West 


2.43. 


1.36 




' 1.90 


Arizona 


1.02 


2.73 




1.88 


Bank street 


1.47 


2.07 




1.77 


Oregon _ 


2.15 


4.36 




3.26 


Kansas • - ■ 


3.04 


.94 




1.99 


High/Scope 


1.30 


0.05* 




.63 


Florida * 


1.94 


3.48 




2.71 


EDC 


3.27 


2.94 




3.11 


.1 

Pit-tsburgh 

\ 


3.61 


2,53 




* 

3.07 


REc\ 


2.81 


.10 




1.45 • 


Bnablers 


.73 


1 A 

. 14 




.44 


Control 








mm 


NPV 


2.43 

* 


3y2i 




2.82 


Column 
Marginals 


2.18 


1.99 




2.08 




F 








Program 


1.933 


.03 






Sex 


.259 


>.500 






Program X Sex 


1.341 


.20 







♦Marginals are unweighted averages of cell means. 
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RESUL'fS OF INTERACTION ANALYSIS 



r 




ETS 

\ 


Program 


V Male 


F3male i 


Par West 


" 

1.97 


1,86 


Arizona 


2.66 


3.24 ;^ 


Bank Street 


1.83 


1.81 ^ 


Oregon 

0 


4 . 09 


2.04 


Kansas 


4. 01 


3 . 50 


High/Scope 


.21 


.24 


Florida 


1.30 


2.60 


EDC 


1 • 66 


1 . 63 


Pittsburgh 


3 . 11 


1 . 88 


REC 


.07 


-.05 


Enaiblers 


-.30 


-.06 


Control 






NPV 


1.42 


1.32 


Coliunn 
Marginals 


1.84 


1.67 




F 


gianif icancG 


Program 


8.032 


<.001 


Sex 


.461 


>.5 


Program X Sex 


.913 


7.5 



\ 



(ROW Margincils* 



1.92 
2.95 
1.82 
3.06 
3.75 

.22 
1.95 
1.65 
2.49 

.01 
-.18 

1.37 



1.75 



♦Marginals are unweighted averages of cell means. 



FRir 



Table VIIl-7 /' 

RESULTS OF INTERACTION ANALYSIS 
PSI 



Program Hhite 




ROW Marginals* 


Par West 


2.89 


1.95 




^•42 


Arizona 


2.03 


1.05 




f 


Bank Street 


.34 


-.34 




n nn 


Oregon 


, 1.79 


2.89 




^ A • OH 


Kansas 


3,58 


1.91 




^ ♦ 


ftigh/Scope 


1.50 


.31 




/ 

* 0 1 


y 

Florida 


1.25 


1.41 




1.33 


EDC 


2.64 


1.66 




2.16 


Pittsburgh 










REC 


2.98 


1.79 


/ 

1 


2.39 






Q R 


i 


1.80 






"J Q 
• J" 


1 

I 


.13 


NPV 


2.11 


1.38 


i 


1.74 


Column 
Marginals 


1.97 


1.28 




1.63 




F 








Program 


4.372 


<.ooi 






Ethnicity 


7.865 


.006 






Program X 


1.009 


.43 







Ethfjlcity ' , 

*Mar9inals are unwuighted averages of coll means. 
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RESULTS OF INTERACTION ANALYSIS 







PPV 




Program 


White 


. Black J 


. ■ 

■Rov^r Marc^lnals* 


Par West 


4.45 


9.65 




7.05 


Arizona 


4.01 


7.38 




5.69 


Bank Street 


3.84 


5.82 




4.83 


Oregon 


-.50 


8.26 




3.88 


Kansas 


4.64 


8.00 




6.32 . 


High/Scope 


2.08 


4.17 




3.13 


Florida 


1.76 


6.86 




4.31 


EDC 


4.69 


6.82 


- 


5.75 


Pittsburgh 


- 


- 


■ / 




REC 


5.21 


8.48 




6.84 


Bnablers 


2.13 


4.67 




3.41 


Control 


4.89 


7.52 




6.20 


NPV 


4.01 


6.19 




5.10 


Column - 
Marginals 


3., 4 3 


6.99 




5.21 




F 


^i.anif icance 






Program 


2.495 


.005 






Ethnicity 


54.395 


<.001 






Program k 


1.392 


.17 







Ethnicity 



♦Marginals are unweighted averages of cell r»eans. 
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OF INTERACTION ANALYSIS 



WRTC 
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v> /V V* r% vVl 




Black J 


x>r%x^j M;^ rrr 1 nn 1 €i* 


Tt^ ^ Thi A M 

ifor west^ 


1 A A 

1 • 44 


1.62 


• 


1.53 




Z • Z / 


2.07 




2 • 17 


Bank StrGGt 


1/37 


1.39 




X • JO 




5.08 


2.74 




^ 3.91 


A ^ v4 ( S «i7 0 


4.29 


4.39 




4 34 


Hloh /ficonp 


3.62 


. 39 




2. 00 


Florida 


2.26 


1.29 




1.77 


EDC. ' 


1.61 


3.09 




2.35 ' 


Pittsburgh 










REC 


.51 


1.19 




.85 


Enablers 


2.25 


1 .56 




1.91 


Control 


.20 






.19 


NPV - w 


2.17 


1.31 




1.74 


Marginals 


2.26 


1.77 




2.01 




F 


— — - — J 

^j.anificance 






Pjroaram 


11.485 


<.001 






Ethnicity 


6.166 


.01 






Ethnicity . 


3.525 


<.001 


* 


* 


v.- 

^Marginals are 


unweighted averages of cell 


hieans . 
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RESULTS OF INTERACTION ANALYSIS 
WRTD 



Program 


, White . 


... .alack j 


( ROW Marginals* 


Par West 


.87 


1*09 




.98 


Arizona 


1.26 


1.03 




1.14 


Bank Street 


.30 


.54 




.42 


Oregon 


2.18 


2.77 




2.48 


Kansas 


% 1.79 


1.94 




1.86 


High/Scope 


.97 


.19 




.58 


Florida 


.65 


.68 


t 


. 66 


EDC 


1.69 


1.16 




1.43 


Pittsburgh 






/ 




REC ' 


.72 


.61 




. 67 


Enablers 


.77 


.60 




.69 


Control 


.22 


.05 




.14 


NPV 


.80 


.85 




.82 


Column ^ 
Marginals 


1.02 


.96 




.99 




P ■ 


^ianificance 






Program 


23.654 


^.001 






Ethnicity 


.564 


.45 






Program X 
. Ethnicity 


1.807 


.05 






*Marginals are 


unweighted averages of cell 


means. 





Table VIIl-ll 
, * 271. 

RESULTS OF INTERACTION ANALYSIS 
ITPA 



Fro<f ram 


wnicB 




ROW rlarg jL naXS" 


Far West 


1.04 


3i02 


2.03 


Arizona 


i .55 


2i20 


f 1.88 






1*82 I 


1.77 


oiregon 




3.84 / 


3.65 




on 


J • 


2.05 


High/Scope 


• 73 


2.38 


.83 


Florida 


-.10 


3.83 


1.87 


EDC 


2.28 


3.40 


2.84 


Pittsburgh 








REC 


♦ 08 


1.07 


.58 


Enablors 


• 56 


.33 


.44 


Control 


- 


- 




NPV 




3.67 


2.90 


Column 
Marginals 
i 


1.17 


2.61 


1.89 




F 


^j.anifloance 




Program 


1.175 V 


.30 




Ethnicity 


6.669 


.01 




Program X 
Ethnicity 


.486 


7 .5 


* 


J' ' 

♦Marginals are 


unweighted averages of coll means. 





ERIC 



Table VIIi-12 



272. 





RESULTS OF 


INTERACTION ANALYSIS 




1* 




ETS 




Program 


White 


Black ] 


l^uw rici tyxiicixo'' 


jTar west! 


X « o*t 


3.74 










3.43 




t 




2.17 


* 




v*Jl on 


1.85 


3.84 


• 


2 85 


Kansas 




4.55 


* 


1 "77 
0 • / r 




.36 


.33 




• 35 


Florida 


.95 


i • 2X 




.1.58 




1.42 


1.82 




1,62 
















. .70 




-.55 


Bnablers 


-.62 


• X 4 




-.38 


Control 












1,10 


1.68 




1.39 


Column 
Marginals 


1.05 


2.21 




1.63 ^ 




r 








Program 


4.023 


.001 






/.Ethnicity 


7.435 


.007 






Program X 
Ethnicity 


.327 


> .5 







^Marginals are unweighted averages o£ cell means. 



.ERIC 



Table VIIl-13. 

273. 



RESULTS OF INTERACTION ANALYSIS 

* ' ' I - - I - - - 

PSI 



Program 


Under 54 




/. Ouar fiO 


ROW Marginals* 


Par West 


3.37 


2.93 


1.07 


2.46 


Arizona . 


2.33 


1.52 


1.25 


1.70' 


Bank Street 


.97 




-1.35 


-.28 


Oregon 


- 


- 


mm 




Kansas 


2. 35 


2.60 


4.18 


3.04 


High/Scope 


1.40 


.58 


1.51 . 


1.16 


Florida 


2.16 


2.68 


.78 


1.6^ 


EDC ' 


1.23 


1.33 


t 2.29 


1.62 


Pittsburgh 


2.96 


1.90 


4.09 


3;oq 


REC 


3.02 


2.27 


-2.45 


.95 


Enablers 


1.75 


1.32' 


. 2.54 


l.fil 


Control 


.29 


.24 


.- . 11 


. . 14 


NPV 


1.95 


1.45 

* 


1.08 


1.49 


Column 
Marginals 


1.98 


3-. 53. 


1.24 


1.58 




F 


??fin?,*;ipancfi. 






Program 


'4.734 


<.001 






Age 


2.586 . 


.08 






Program X 
Age 


2 . C66 

• 


.003 







^ ■ 

^Marginals are unweighted averages of cell means » 

. • ■ ■ < 



Table VIIl-14 ' 
^ 274. 

RESULTS OF INTERACTION ANALYSIS 
PPV 

» t 

#* . 



Program 


Under 5,4 


54-60 


/ Ovier 60 


ROW Marginals* 


Par West 


5.72 


6.84 


2. 34 


4.97 


Arizona 


4.78 


5.51 


5.23 


5.17 


Bank Street 


5,75 


6.42 • 


4.64 


5. GO 


Oregon 






mm ' 




Kansas 


8.61 


5.13 


4.22 


5,99 


High/Scopo 


3.69 


5.46 


1.24 


3.46 


Florida 


7.49 


7.98 


3.97 


6.48 


EDC 


5.31 


5.78 


6.60 


5.90 


Pittsburgh 


7.71 


3.84 < 


• 2.79 


4.78 


REC 


13.57 


6.56 


10.97 


10.37 


Erablors 


4.41 


5.32 


3.05 


4.26 


Control 


6.67 


4.18 


6.57 


5.81 


NPV 


6.47 


5.96 


6.71 


6.38 


Column 
Marginals 


6.68 


5.75 


4.86- 


5.76 


i 


F 




V 




Program , \ 


2.819 


.002 




V , ' 

r 


Age 




3.249 


.04 






.. Program X 
Age ■ 




1.015 

4 


.44 







^Marginals are unweighted averages of cell weans. 




Table viii-15 

275. 





RESULTS OP INTERACTION ANALYSIS 








WRTC 


• 




Program 


Under 54 


54-60 




• ■ 

pow Marginals* 


Par West 


2.07 


1.29 


• 1.57 


1.64 


Arizona 


1.13 


2.28 


2.76 


2.P6 


Bank Street 


.26 


1.12 


2«92 


1.43 


Oregon 






> 




Kansas 


3.75 


4.71 


5.35 


4.61 ' 


High/Scope 


.30 


1.05 


5.34 


2.23 


Florida 


1.46 


1.95 


2.09 


1.84 


EDC 


2.80 


3.36 


2.48 


2.80 


Pittsburgh 


1.15 


1.64; 


2.99 


1.93 


REC 


1.38 


1.33 


.81 


1. 17 


Enablers 


.1.24 


1.^6 


3.28 . 


1.95 


Control 


.13 


1.26 . 


.13 


.51 


NPV 


.42 


1.45 


3.17 


1.68 


Column 
Marginals 


1.34 


i;9o 


2.73 


. 1.99 




F 








Proaram 


8.387 


^.001 






Age' 


16.284 


<.ooi 






Progreim X 


2.437 • 


^ .001 




• 



^Marginals are unweighted averages of cell moans. 

ERIC . . . . 



Table VIII-16 



RESULTS. OF INTERACTION ANALYSIS 
WRTD 



276. 



ProgrAJn 


Under 54 . 


„ 54-60 


/ Over. 60 


ROW Marginals* 


Par West 


.86 


.93 . 


.87 


.89 


Arizona 


• 82 


1.24 


1.30 


1.12 


Bank Street 


• 21 


.41 


.92 


.51 


Oregon 




- 


- ■ 


- 


^ Kansas 


1,96 


1.89 


1.23 


1.69 


High/Scope 


.25 


.62 


1.17 


.68 


Florida 


.70 


\64 


.67 


.67 


EDC 


,75 


1.18 


1.49 


1.14. 


Pittsburgh 


1.07 


1.01 


, 2.19 , 


1.42 


REC 


.73 


.68 


.49 


.64 


B^ablers / 


.43 


.74 


.92 


.70 


.Control 


. .09 


.20 


.56 


. .28 


NPV 


.39, 


.69 


1.26 


.78 


Column 










^^Marglnals ' 


. .69 


.85 


li09 


.88 




F 






• 


Program 


8.332 


■ . 

<.001 






Age 


8.382 


<.001 






Program X 


1.588 


. .04 







^Marginals are unweighted averages of cell means. 



Table VlIl-17 

' ' . 277. 

■ ■ . • -••■)• 

RESULTS OF INTERACTION ANALYSIS 



Program 


No Prior, 
Preschool 


Prior 
. Preschool 


i 


l^ow Marginals* 


Par West 


2.82 


2.03 




2.43 

1 


Arizona 


2.18 


.40 




1.29 


Bank Street 


. 37 


-.93 




-.28 


Oregon 


3.63 


.87 




2.25 


Kansas . 


3.02 


-2.77 




.12 


High/Scope 


1.26 


.37 




,81 


Florida 


1.77 


2.39 




2.08 : 


EDC 


1.55 


2.21 , 




1.88 


Pittsburgh 


3.06 


.77 




.192 


REC ^ 


2.43 


2.78 




2.60 


Enablers 


1.77 


2.39 




2.08 


Control 


.54 


-.58 




-.02 


NPV 


1.80 


.54 




1.17 


Column 
Marginals 


2.01 


.64 




1.32 




P 








Program 


4.699 


<.O01 / 






PSEXP 


31.179 


(»001 






Program X 
PSEXP 


3.598 

f ■ 


<,001 






♦Marginals are unweighted averages of cell 


irteans. • 


f 


ERIC 






** 

1 


1 1 I" 



Table Vlll-ld 



\ , 




4 


.278. 


1 / • 

{ 


RESULTS OP 


INTERACTION ANAIiYSIS 








pp'v 




■ 1 

Proqfam 


No Prior 
Preschool 


Prior 
Preschool 


' ^ ' (ROW Marginals* 


Par West 


6.07 

* 


3.84 




4.96 


Arizona 


5.60 


4.37 




4.99 


Bank Street 


6.32 


4.59 




5.46 


Oregon 


8.28 


4 .00 




6.14 


Kansas 


6.82 


3.73. 




5.28 


High/Scope 


3.63 


3.28 • 




3." 4 6 


Florida 


6.28 


1.98 




. 4.i3 


EDC 


5.17 


7.13 




6. 15 


Pittsburgh 


6.39 


3.82 


/ 


5.10' 


REC 


9.71 


11.81 




10.76 


Enablers 


4.91 


Q 1 






Control 


7.04 


3.40 


t 


5.22 


NPV 


6.78 


5 . 18 




5.98 


Marginals 


6.38 


4.47 


V 


5.43 




F 


Sj.qni,€iftance- 






Program 


3.890 


<.001 


) ■ 




PSEXP 


~ 13.216 


<.001 


/ 


t 


Program X 


1.255 


.24 







PSEXP 

♦Marginals are unv/eighted averages of cell means. 



' . Table VIlI-19 

279. 

RESULTS OF INTERACTION ANAl4,VSI8 
WRTC 



\ 

I 



Program 


Preschool 


frior 
1 Preschool 


1 (ROW Marginals* 


Par West 


1.54 


1.35 




1.45 


Arizona . 


2.03 


2.46 


> 


2.24 


Bank Street 


1.25 


1.55 




l.'JO 


Oregon 


3.56 


2.45 




3.00 


Kansas 


4.53 


2.29 




3.41 1 


High/Scope 


>' 2.5 3 


1.20 




1.86 


Plori<ia 


1.87 . 


2.41* 




2.14 


EDC 


3.14 


2,32 




2.73 


•Pittsburgh 


, 1.64 ■ 


.74 




.1.19 


REC 




1.14 




1.26 


Enablers 


1.78 


■2.81 






NPV 


.65 
1.93 


-.56 
1.54 




fit* 
1. 74 


Column 
Marginals 


2.14 


1.67' 




1.90 


■ 


F 








t^rbgram 


5^452 


(.001 






P^EXP 


5.054 


.025 






Program X 


1.449 


.14 







PSEXP ' 

f 

♦Marginals are unweighted averages of cell means. 



table VIIt-20 



280. 



RESULTS OF INTERACTION ANALYSIS 



1 




WRTD 


$ 




Progi^m 


No Prior 
Preschool 


Prior 




\ (ROW Marginals* 


Par West 


.93 


.75 






.84 


Arizona 


1, 15 


1.17 




1.16 


Bank Street 


.41 


.61 




.51 


Oregon 


2« 81 


2.08 




2.44: 


Kansas 


1.93 


1.32 




1.G2 


High/Scope 




.82 




.76 - 


Florida 


• / •* 


.05 




. 40 


EDC 


1.02 


1.53 




1.27 


Pittsburgh 


1.21 


.70 




.96 


REC 


.61 


1.17 


- 


.89 


Enablers 


.72 


.66 




• 07 


Control 


18 


.11 






NPV 


. 82 


.93 




.87 


Column 
Marginals 


1.02 


.91 




.97 




r _ 


gionif icance 






Program 


16. J. 37 


<.Q01 






PSES^P 


1.641 


.20 






Program X 


2.166 ■ ' 


.01 







PSEXP 



^Marginals are unweighted averages of cell means. 



ERIC 



Table VIII-21 



RESULTS OF INTERACTION ' ANM«YS 18 
* ITPA 



Program 



No Prior 
Preschool 



Fai: West 

Arizona 

Bank Street 

Oregon 

Kansas 

High/Scope 

Florida 

EDC 

Pittsburgh 
REC 

Enablers 

Control 

NPV 



Column 
Marginals 



Program 

PSEXP 

Program X 
PSEXP 



1.96 
1.55 
1.49 
3.06 
2.10 
.7.0 
2.43 

3.57 
3. 87 
1.69 
.41 

2.67 



2.13 



Prior 
Presghool 



1.24 
2.43 
2.20 
3. 31 
2.17 
.60 
3.19 

2.67 
.20 
.22 
.53 

3.43 



1.85 



1. 192 
.304 
.560 



Sjgnif icance 



.29 

> .5 

> -5 



♦Marginals are unweighted averages of cell means. 



Table VIII-22 > ' 

282. 

RESULTS OF INTERACTION' ANALYSIS 





* 


ETS 


c 

/ 


Program 


No Prior 
Preschool 


Prior 

Preschool j 


ROW Marginals* 


Par West 


1.79 


2.53 




2.16 


Arizona 


3.32 


1.87 




2.60 


Bank Street 


1.95 


1.70 




1.82 


Oregon 


3.19 


3,30 


• 


3.24 


Kansas 


3.86 


3.40 




3.63 


illgh/Scope 


.12 


.63 




.38 


Florida 


1.61 


3.09 




2.35 


EDC 


1.61 


1.68 


« 


1.65 


Pittsburgh 


2.42 


2^92 




2.67 


REC 


-.10 


.36 




.23 , 


bnaoiers 


-.01 


-1.00 




-.51 


Control 










NPV 


1.53 


1.20 




. 1.37 


Column 
Marginals 


1.77 


1.82 




1. 80 




F 


5j.qnj,f icancp 


f 




Program 


4.858 . 


(.001 






PSEXP 


. 021 


> .5 






Program X 
PSEXP 


.501 






r • 








A 





^Marginals are unweighted averages of cell means. 



Table VIII-23 



283. 



RESULTS OP INTERACTION ANALYSIS 



/ 



a 




PS I 


* 




•IS >S M A «M 

progrcuii . 


Under 10 


10 or 11 


Over 11 


(ROW Marcfinals* 




4.07 


2 . 57 


2.43 


3.02 


c 


2 10 


1 11 


1 36 


1 66 

X . w 




*-.15 


-•25 


-•26 


-.22 


v.ii ecjpn 


4 ii 


3.07 


2 * 12 


3 . 10 


A M O & d 

JxcinSaS 




1 63 

X a V ^ 


2^57 


2 47 




1.82 


1.21 


.43 


/ 1.16 


FJLonaa 


^ 2 01 


2 21 


26 


1.49 


PHP 


2 .97 


1.40 


.67 


1.68 


« Ah a* ^« • « %A V« 

FictSDUirgn 


2 .96 


2.51 


2 . 50 


2.66 


RPC , 


2.05 


3.03 


2^26 


2.45 


Enablers 


3.. 65 , 


1.46 


•96 


2.02 


Control 


. .76 


-.42 


.31 


.22 


NPV 


2.27 . 


1.11 


.75 


1.37 


Marginals 


2.46 


1.60 


1.26 


1.78. 




F 


^loniflcancG 






Program 


8.578 


4.001 






Mother ' s 
Education 

Pjrogjram . X 

Mother's 

Education 


14.160 

' .97 


< .001 




* 


♦Marginals are unweighted] averages of cell 


uieans « 




ERIC 


1 


f 




1 



Tablia VIII-24 * 
0. 284. 



RESULTS OF INTERACTION ANALYSIS 

" "# - . I " . ' 

PPV ^ " 



Program 


L. , Uadgr XO : ,, 


, 10 n,r 1 1 


f nvpr 1 1 — 1 


(ROW Marginals* 


Par West 


9.00 


6.00 


4.53 


6.51 


Arizona 


6.40 


4.04 


5.29 


5.24 


Bank Street 


5.33 


5.74 


5.51 


5.53 


Oregon 


10.88 


7.87 


4,17 


7.64 ' 


Kansas 


7.78 


6.20 


5.77 


6.58 


High/Scope 


6.43 


3.92 


.93 


3.76 


Florida 


5.44 


6.79 


5.47. 


5.90 


EDC 


7.58 


5.37 


4.70 


5.88 


Pittsburgh 


6.94 


9.48 


' 4.14 , 


6.85 


REC 


13.16 


9.93 


8.56 


10.55 


Enablers 


8.62 


4.00 


1.44 


4.69 


Control 


8.83 


5.81 


4.33 


6.34 






R on 




C 11 
D . O O 


Column 










Marginals 


7.96 


6.23 


4.69 


6.29 




F 


^j.anificahce 






Program 


4.864 


< .001 






Mother's 


21.626 


<.001 






Education 










Program X 


1.354 


.12 







Mother's 
Education 



♦Marginals are unweighted averages of cell means. 



Table VI I I -2 5 



'285. 



RESULTS OF INTERACTION ANALYS IS 
WRTC 



Program 



Fa^r. West , , 
Arizona ' 
Bank Street 
Oregon 
Kansas 
High/Scope 
Morida 
EDC 

Pittsburgh 
REG 

« 

Eriablers 
Control • 
NPV 



2.05 
3.06 

.•live 

3.7'6 
<. 37 
'2.38 
' 2.41 
3.13. 
\ 2i31 

■ / 

2.42 
.48 
2.26 



IQ or 13, 



/ .Qvfir 11 



1.18 
1.67 
, 1.02 
3 -.3 6 
3.68 
2.0.4 
.2.12 
2.52 
1.16 
1.89 
1.81 
.88 
1.78 



1. 61 




1.61 


1.91 




2.21 


1.44 




1.41 


2.97 




3.36 


4.63 




4.23 


2.54 




2.32 


.80 




1.78 


2.31 




, 2.65 


1.37 




1.62 


.96 




1.33 


1.71 




1.98 


.16 




.49 


i.n 




1.92 



ROW Margi n als'* 



Column' . 
Marginals 



.2 ,43 



1.93 



1.85 



2.07 



Prograin . 

Mother' s 
Education 
Program X 
Mothered 
Education 



?.jq,n;i.^icange 



10.662, 
5.027_ 
.664 



<.001 
.01 



j^I^Qlnals dr^ unweighted averages of cell nteans< 



Table ,vi 1 1 -2 6 

286. 

RESULTS OF INTERACTION ANALYSIS 

. . - 

WRTD 



I 
I 

! 





Under 10 


10 or 11 


' , vYgC All , , ■ . 1 


iRow Maroinals* 


Par Vtest 


.79 


.68 


1. 11 




Arizona 


1.38 


1.09 


1.05 


1 18 


Bank Street 


.78 


.37 




CO 


Oregon 


2.76 


2.95 


2* 4-3 


■5 

71 


Kansas 


, 2.02 


1.49 


1.94 


1 fl2 


Hicf h/Scot>G 




. 59 


• 75 


.71 


Florida 


.71 


.71 


.56 


.66 


EDC 


1.57 


1.15 


.90 


1.21 


Pittsburgh 


,92 


.90 


1.2< 


1.02 


REC 


.78 


• 85 


.53 


.72 


Enablers 


.81 


.63 


.71 


.72 


Control 


.13 


-.13 


,39 


.13 


NPV 


. 85 


.78 


.93 


.85 


Column 










Marginals. 


1.10 


.93 


l.OC 


1.01 


Program 

Mother's' 
Education ^ 
Program X 


f 

P 


^j.anif icance 






« 

10.662 
5.027 
.664 


< .001 

.01 
>.5 



Mother's ' 
Education 

ERXC?^***^® are unweighted averages of cell means. 



Table VIII-27 



RESULTS OF INTERACTION ANALYSIS 

T " ' 

ITPA 



287. 



ProqraiTi 



Under 10 



iq.gr 11 \ L ■■ Qv^r, -11 



[ROW Marginals* 



Par West 
.Arizona 
Bank Street 
Oregon 
Kansas 
High/Scopfe 
Florida 
EDC 

Pittsburgh 
REC 

Enabl^rs 

Control 

NPV 



Column 
Marginals 



Program 

Mother's 
Education 
Program X 

Mother-* 8 
Education 



3.57 
3,01 
2.33 
2.38 
3.21 
.58 
1.03 
.3.23 
3.90 
-.96 
1.65 

1.84 



2.15 



1.644 
.278 
.966 



.60 
.91 
1.92 
2.57 
3.71 
1.70 
3.16 
1.98 
2.2 8 
.87 
-.45 

3.38 



1,89 



Significance 



.08 

> .5 

> '5 



2.10 
1.8< 
1.38 
3.tf9 
.48 
.11 
3.56 
4.08 
3.05 
2.66 
.55 

3.32 



2.25 



2.09 
1.92 
1.87 
2.95 
2.47 
.79 
2.58 
3.10 
3.08 
.86 
.58 

2.85 



2.10 



ERIC are unweighted averages of cell means 



TaWe Vili-28 



288. 



results of interaction analysis 
ets 



Program 



Under 10 



10- or 11 / Over 11 



Far West 

Arizona 

Bank Street 

Otegon 

Kansas 

High/Scope 

Florida 

EDC 

Pittsburgh 
REC 

Enablers 

.Control 

NPV 



Column 
Marginals 



Program 

Mother's 
Education 
Program X 
Mother's 
Education 



1.55 
3.22 
2.84 
3,81 
3.03 

.99 
2.03 
2,10 
2.54 
-.65 

.31 

1.52 



1.94 



F. 



8.304 
6.184 
.732 



3.04 
3.63 
1.38 
4.34 
5.59 

.61 
3.20 
1.88 
2.88 

.24 
-.54 

1.68 



2.83 



ROW Marginals* 



1.34 
2.27 
1.43 
2.05 
3.31* 
-.37 
.54 
.57 
2i40 
.13 
-.19 

.89 



1.21 



< .001 
.003 
> .50 



1.97 
3.04 
1.88 
3.40 
4-. 00 
.41 
1.92 
1.51 
2.61 
-.09 
-.14 

1.36 



1.82 



?/-ginals are unweighted averages of cell means, 
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kapter IX 



MAJOR CONCLUSIONS 



Throughout this report we have focused on three major 
questions. Each of our four analytical approaches has 
provided evidence bearing on these questions. Iii this 
chapter we summarize the evidence and, present conclusions* 

IV' To what extent does a Head Start experience 

accelerate the rate at which disadvantaged 

< ■ ' ' ' ■ ■■ 

pre-schoolers acquire cognitive skills? 

bur evidence here is of two types. Each analysis 
provides a ^coir^arison between the performance of Head 
Start ahd Control children for the six tests taken by 
, both. The residual analysis provides a direct estimate of 
.the amount of growth attributable to Head Start over and 
above lih at the child would otherwise have achieved. On 
the Preschool Inventory (PSI) ^ WRAT Copying Marks (WRTC) , 
WRAT Recognizing Letters (WRTR) , WkAT Naming Letters (WRTN) , 
and WRAT Reading Numbers (WRVD) , the Head Start children 
(both PV and NPV) did substantially better than the Control 
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Children. On the Peabody Picture Vocabulary Test (PPV) 
Head Start and Control performances were comparetble. 

From the residual analysis, we found that the growth 
rates for Head Start children oh all six tests considered 
increased substantially. For the PSI, the growth rate 
increased by about ^0%,, for the PPV by about 100%, for the 
WRTC 200%, for the WRTD 300%. For the .ITP A Verbal Expression 
the gain was approximately 100% i and for the ETS Enumeration 
about 75%. Moreover I except for the PPV, the average 
residuals for the Controls were near zero. For the PPV, 
the average residual for the Controls was close to that 
for both PV and NPV Children. In conclusion it seems fair 
to say that: 

In terms of a wide variety of cognitive skills. 
Head Start is effective in accelerating the growth 
rate of disadvantaged preschoolers. 

We do not know, of course, whether the changes 
wrought are permanent and can be built upon. Has Head 
Start singly made the child a bit more aware of certain 
specific things at a particular point in his life, or 
has it altered him more profoundly and increased his 
capacity to learn. The answer is probably to some ex- 
tent unique to each child. 
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There Is perhaps a certain pessimism at present about 
our ability to effect desirable social change. Head Start 
Is only part of a child's life experience over a short 
period of his life. If even this relatively minor effort 
to alter the child's environment can have substantial, 
measurable litpacti then there Is reason to hope that more 
extensive societal efforts may have profound and lasting 
effects. 

2. Are Planned Variation models, simply by virtue 
of sponsorship, more effective than ordinary 
non^-sponsored Head Start progx^ams? 

There are two reasons why we might expect PV programs 
to be generally more effective than NPV programs. First; 
they involve the expenditure of substantially more money 
per child. The nature of these expenditures is detailed 
by KcMeekin (1973). Second, we might expect a great deal 
of effort on the part of sponsors to ensure that their 
approaches perform optimally. In fact, if we were to find 
PV programs generally superior to NPV programs, we might 
be concerned that this was the result of special effort 
expended in the competitive experimental situation which 
might disappear when the programs were routinely imple- 
mented. 
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Smith (1973) found no overall difference in perforrwrnce . 
between and NPY prograiAS for the 1970-71 cohort. Our 
analyses strongly support this finding. There are no 
clear differences between the 28 PV and the 12 NPV sites on 
any test. The general picture which emerges is that: 

Relative to the condition of no preschool program, 
the effects of Head Start programs are quite homo- 
geneous, with no systematic differences between 
sponsored and non-spon§ored programs. 

3. Are some PV models particularly effective at 
imparting certain skills. 

We have results for each of eight tests in our 
battery on at least three of the four analyses. Table IX-1 
presents an overall summary of inter-model comparisons. 
Each analysis has its own assumptions and implicit or 
explicit measure of program effectiveness. Thus we would 
not expect the different analyses to yield identical 
results. We would, however, be concerned about large ^ 
discrepancies. An effect which shows up consistently 
over several analyses is more likely to be real, arid not 
simply an artifact of spme mathematical manipulations. 
In Table IX- 1 we have given a model + on a particular 
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test if it achieves -i- on at least two of thr^e analyses. 
We have given a ++ only for models with at least + on 
all analyses and at least one ++. The same standards apply 
to negative effects. These standards are of course 
arbitrary, and the reader is free to apply his own stan- 
dards to summarize the results in Chapters IV through VII. 

Smith (1973) foimd a ratiier small number of exan?>Ies 
of programs which were especially effective at i>rcmoting 
skills. He also reached the tentative conclusion that 
"differential model effects are more easily discerned if 
the outcome measure taps specific rather than general 
cognitive growth." Our results tend to corroborate these 
findings. Only 22. "effects" are cited for the eight tests. 
Three of our tests (WRTR, WRTN, WKTD) measure very specific 
academic skills. Two others (WRTC, ETS) measure skills 
which are somewhat more general but relatively easily 
taught* Three tests (PSI, PPV, ITPA) measure general 
skill relatively difficult to teach in a preschool pro- 
graun. These three account for only 6 of the 22 effects. 
Moreover, 3 of these effects are on the ITPA, which in 
terms of reliability and validity is the most questionable 
test in our battery. The PPV, which is probably our 
most general measure, shows no effects. The 32 -item PSI 
we have used does appear to be possibly more sensitive to 



294. ' 

program differences than the 64*-ltem version used in 
1970-*71« The clearest and most dramatic oxey^les of 
special program effectiveness are Kansas on the WRTC and 
Oregon and Kansas on the WBTD« 

We mentioned In Chapter I the hypothesis that the 
•^academic" models (Oregon, Kansas, Pittsburgh) , which 
con8clous3.y eirphasize the acquisition of academic skills, 
would be overall more effective: than the other models • 
Prom Tabl*! IX-1, we see that of the 17 positive effects 
noted, 12 are for these three models. Moreover, none of 
the three received a - for any test on any of the analyses. 
Kansas hajj four ++'8 and Oregon three. The only other 
model which can lay claim to better than average overall 
performance is Arizona, with three ppsitive effects and no * 
negatives, Our conclusions "in tejrms of inter-model 
comparisons can be summarized in the following statements. 

Head Start programs^ are quite homogeneous in 
their* ability to promote general cognitive development. 

Ho Head Start program is of above average effec- 
tiveness for all of our measures. 

Oregon and Kansas appear to be overall particularly 
effective in inparting specific academic skills. 



a) 



b) 



c) 
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4) Arizona and Pittsburgh mav be overall particularly 
effective in imparting specific academic skills. 

e) No progjL"£um appears to be overall particularly 
ineffective* 
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Table IX-I 

OVERALL SUMMARY OF RELATIVE MODEL EFFECTIVENESS* 



++ Indicates model appears to be highly effective. 
+ Indicates evidence for above average effectiveness. 
Indicates evidence for below average effectiveness. 
Indicates iTK)del appears to be liighly ineffective. 

I ■ . ■ ■ 



Model 


PSI 


PPV , 


WRTC 


WRTR 


WRTN 


WRTD 


ITPA 


ETS 


Par West 


+ 
















Arizona 








+ 


+ 






++ 


Bank Street 


















Oregon 


++ 




+ 




+ 


++ 




++ 


Kemsas 






++ 






>+ 




++• 


High/Scope 


















Florida 


















EDC 


















Pittsburgh 






r 


+ 






+ 


+ 


Enablers 



















*REC not included because with only one site we felt it ui^fair 
to draw any conclusions. 
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DESCRIPTION OF VARIABLES 

Thi a appendix aescribes the child, classroom, teacher, 
and outcome variables used in the preceding uhalyses. Where 
multiple forms of essentially the same variable existed, 
they are noted. Variable nmea as referred to elsewhere are 
capitalized. Where categories are given as unindexed lists 
tf9hite/)Black/Mexican-American) , the codes were the ascending 
integers 1, 2, 3, etc. Related variables are grouped together 



I. Child characteristics, dem<>graphlc and background . 
(Source: Classroom information Form) 



AGE 
A6E1 

AGED 



Child's age In months as of October 1, 1971 
for each test, child's age in months as of 
fall test date. 

for each test, interval in months between 
fall and sprijn^ testing. 

Note : a separ'ate value of AGEl and AGED was 
computed for each test, e.g. AGEl (6) for the 
WRAT-D. 



DAYSAB 



days absent during the Head Start year 



ETHWHITE White ethnicity. l«white, O»not white 
ETHBL Black ethnicity. . l«Black, 0«»n6t Black 

Note : for the residual analysis sample, 
children other than White, Black, Mexican- 
American and Puerto Rican were deleted. 



FLANG 



first language spoken in the child's home. 
0=Engli8h, l»not English 
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PSBXP preschool experience, some non-Head Start/ 

none/some Head Start 
PS preschool experience recede. l»sojne# O»none 

P8MNTHS months of Head Start preschool experience 

if PS5XP was 3 



SEX child *s sex. l^female, 0"male 



II. Child Household Characteristics 

(Sources Classroom Information Form - 
SBXHH, HHSIZE 
Parent Information Form 



- HOMED, FAMINC^ 



FAMING annual family income in $100* s. 

0-98, or 99 if $9,900 or more 



HHSIZE 



total number of persons resident in household 



HOMED mother's education in school years. 

0-16, or 17 if any graduate work 



PIFl 
PIP2 

PIP3 



child watches Sesame Street. Inyes, 0«no 
how often each week. 5+ times/4 or 5/2 or 3/ 
1 or less 

parents also watch, always/usually/sometimes/ 
hardly 



PIF4 materials available for child at home. l»yes, 

to 0»no. blackboard, chalk, colored paper, scissors,. 

PIPI? crayons, coloring books, paints, clay, other arts 

and crafts, music equipment, alphabet/number cards, 
gameSf. puzzles, children's records respectively. 



parents read to child. l«yes, 0«no 

how often per week, less than once/once/several 

times/daily 
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PIF20 how far parentu wish child to go In school • 

High School/Oollege/Graduate School , 

PZF21 how far parents expect child to go Jn school* 

Same 



PIF22 household attributes* l»present, O»not pr^ifent 

to autOr black and white TVr color TV, encyclopedia^ 

PIF30 dictionary f clothes washer, vacuum, hifl, tele-- 

phone respectively 



PIF40 child likes Head Start, very ifiuch/some/not at all 

PIF41 parents satisfied with Head Start, yery/f airly/not 



SEXHH sex of household head, father if present. 

l»feniale, 0«rnale > 



III. Classroom and Teacher Characteristics (non-control children) 
(Source: Teacher Information t'orm 
Rating Forms 



CYTPE 



classroom type. PV/non-PV/control 



SITBI SRI site code 

8ITE2 Huron site code ' 

SPONSOR site sponsor. 2-27«PV sponsors, 2 8«control/ 
30«non-PV 



TIF4 
TIF5 



fall implementatioh rating of PV classroom 
by sponsor. 0-9, 9 highest 
spring same 



TIF6 
TIP7 



fall rating by site director. 0-9, 9 highest 
spring same ' 
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TIF8 clasB watches Sesame Stre^dt, l^yesi 0«no 

TIF9 how often« days per week 

TXPIO nujnber of field trips taken* 0-10, or ll^inore 

than 10 



TIFll teacher sex* l»niale^ 0>»feinale 

TIP12 teacher ethnicity* AM* Indian/Black/Oriental/ 

White/Mexican-American/Puerto Rican/Cuban/ • 

other Spanish/Posftuguese 
TIF13 teacher marital status* single/marified/widowed- 

divorced-separated 
TIP14 teacher has children* l"yeO| 0-no * 

TIF15 teacher neighborhoood similar to center. I»ya8, 

Owno 

TIP16 teacher age* years 

TIP17 teacher education* 0^16 years i or 17«more than 16 

TIF18 teacher certification* none/temporary/regular 



TIF19 teacher rating of classroom differs from 

sponsor's model* much/sme/none 
TIF20 sponsor changed teacher's ways* very much/much 

TIF21 teacher use of model given choice* use/change 

some/change most/not use 



IV* Outcome Measures * 

These are more fully described in the body of the report 
and in an earlier report, *The Quality of the Head Start 
Planned Variations Data"* Tester assigned validity was 
used as a criterion for accepting a score. 



PSI Preschool Inventory 



PPVT Peabody Picture Vocabulary Test 



ERLC 



WRAT 
C/R/N/D 



Wide Range Achievement Test Copying marks/ 
Recogiiizlng Letter s/Kmming Letters/Reading 
Numbers subtests respectively 
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Illinois Teat of Psycholinguistic Abilities 
Verbal Expression Subtest 



Educational Testing Service Enumeration Tc t 
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Appendix B 
SITE WEAN RELIABILITIES 

In this appendix ve explain the basis for the estimates 
of site-mean reliabilities quoted in Chapter IV. Shayoroft 
( 1962) provides a formula for the reliability of the means 
of groups of size n which may be written as 

, (l-ri ) 

G nB - < > 

■I ^ ' 

whdre and Tj are the reliabilities for the groupa and 

for indlvldualsr and B is the proportion of ;lndlvldual teat* 

variance which lies between groups. We have carried out an 

unweighted-means analysis of variance ^^ith sites as factors 

for each of our eight tests. The proportion of. variance 

which is l^etween sites and th,b harmonic mean n* of the 

site san^le sizes have been recorded in Table C-lt 

A useful and convenient way of summarizing the Information 

provided by Shayoroft *s formula for our tests is in terms of 

the quantity 

1 - rj nB 

We ^an think of y as the proportion of the gap between rj and 
1 which is closed by aggregation to the group level. Let y* 
be the value of y for sites ofoiize n*. Then y* provides a 
^ rough idea of the improvement afforde^ by aggregation to the 
£Bi£iite level for each of our tests. These Values are also displayed 



Table B-1 
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Information Used to Estimate Site Mean Reliabilities 



Test 


B 


n* 




PSI 


22.67 


77 


.94 


PPV 


19.93 


80 


.94 


WRTC 


15.63 


78 


.92 


WRTR 


9.43 


78 


.87 


WRTM 


4.37 


■^8 


.71 


WRTD 


6.79 


78 


.81 


ITPA 


26.55 


38 


.90 


ETS 


28.78 


36 


.90 
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This . appendix presents graphs of fall and spring 
test scores versus age for each of the ei^ht tests in 
oiur battery. The sample is broken out into children 
with no prior preschool experience and those with soma 
prior preschool esqperience. 
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\-miiT COPYIIIG rARI^S SUBTEST FOR ALL CHD 
WITH KO PRIOR PREr^SCHOOL EXPERIEIICL 



Pall 
Spring 



8 



40 

.) 

re 4 

'eric 



19 




46 



52 k 



SB 



64 



70 



76 



68 221 397 405 373 249 146 157. 107 50 2 
8 6 22 94 302 323 322 308 107 141 125 



82 



95 12 
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Mean 
Score 

12 



10 
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WWVT COPYIITG ?'ARKS SUBTEST FOR ALL 
CHILDREN ITITH PP.IOR PRE SCHOOL CXPERIEllCE 



Fall 
Spring 
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40 

) . 
re 
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77 


97 


73 10 
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I He an 
Score 
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WRAf REC0 GNI2IHG LETTERS SUBTEST FOR ALL CHI LDREN 
WITH HO PRIOR PRE-SCHOOL C^tPERIEKCE 

rail : 
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WRAT HAVING L ETTERS SUBTEST FOR ALL CHILDREN WITH 

• MO PP£*-SCH6<:)L bXt>Eftlbt:CE 



Fall 



Pprihg — 



-V 




68 221 397 ^.03 373 249 14G 157 107 50 
0 6 22 94 302 323 322 308 1G7 141 



125 
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34 64 62 79 119 98 92 95 
20 44 60 €0 90 112 
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WR AT READIMG NOSERS SUBTEST FOR ALL 



Spring 
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ITPP SCOPJ^P — FOP. ALL C!III.DRi:iI V^ITH 
^i^RlClR i>Pt-SCHCCL EXPERIENCE 



Fall 



. Spring 



Mean 
Score 



2f 



2C 



15 



IC 



e 

St 



40 




46 




52 




58 




64 




70 




76 


82 


7 


18 


37 


22 


26 


46 


41 


39 


29 


14 


1 










11 


25 


28 


20 


35 


42 


31 


37 





ERIC 



BTS SCORBg- 'o y ALL 
rvtTfr MO t>feIOR- I>R£"i;CHC^L MXPERIETfCS 

Fall 

Spring 




46 52 58 64 

21 77 132 162 145 86 54 73 
4 3 11 29 116 133 137 122 
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Appendix D 
THEORY OF RESIDUAL ANALYSIS 

The purpose of this appendix is to ^ive more theoreti- 
cal underpinning to tK^ residual analysis described in 
Chapter V by describing explicitly the mathematical model 
on whiV:h it is 'based « 

Let and Y^^ • represent the obseirved pre and post 
test scores for individual i in group j. Let Tj^j and Tj^^' 
be the corresponding true scores. We assume that 



'ij> - V-<^lj'> e 



V(e..) - V.(B..'> - 0^^. V I, J 



Let a^^ j and a^ j ' be the age of individual i in group j at 
pre and post testing times respectively. Let M^j be the 
component of true score representable as a linear function 
of measurable variables other than age. Let S^^^ be the * 
component of true score which is independent of both age 
and other measurable variables. 

We assume E(S£j} is constant for all individuals in 
the same treatment group. Let 



326 



.1 



If treatment groups do not differ In terms of average true 
score unrelated to measured variables (as would happen r 
for example, if individuals were assigned randomly to treat- 
ment groups)/ then §"j is 0 for all groups. Otherwise may 
differ from 0 for some or all groups. 
Let 

Our basic model can be represented by the following 
equations: . 

. . T^^ = a + ea^. + + S^^ 

where r^ represents a residual effect attributable to the 
program to which the child is exposed between the pre and 
post tests. Then 



^ " ''ij' ^ ^ij - 'ij 

where* A^^ = ^^*ij' ^ij^ 

For any individual^ a sample residual can be computed 
in either of two reasonable ways: 
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»e can Interpret ^^^^ the observed gain minus the 
expected gain, and rj^j2 as the observed poet-test score 
minus the predicted post-test score. Note that 

; ijl j i3 , ij 

^ ^ ^. = r, + S + e • 

id2 j j ij 

So that 

J iji e 

^J"^ 3 J ij2 e s 

Thus for any individual, r^^^^ is an unbiased estimate of 

r^ with variance 2o^^, and r. has bias S'j and variance 
j e ij2 3 

a^^ + o A useful measure of the accuracy of an estimator 
is the mean sguared-error (MSE) , which equals the sum of 
the variance and the square of the bias* .Thus, we have 

MSE(r^^3^) « 20^^ 

MSE{r^j2) « + ^2g + (^^)2 

Suppose now that we consider combined estimates of 
the form 

r^j « wr^ji + (1 - w)r^j2 1 ^ 1 1 

Then 

. E(r . .) = wr. + (1 - w) (r, + ^j) 
so that the bias is {1 - w)2'^. 
ErJc V(fij) = wV(?i^^) + (1 - w)2v(?^.2) + 
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But 



Thus 



and 



V{r^^) « o^^{2w2 +.w2 + 2w<l-w) } + (1-w) ^(y^^ 



MSE(r^j} - a2^{l+w2} + (l-w)2{(g'^)2 + qZ^j 



Minimizing this with respect to w, we find 



e 



a2g + (S'j)-^ + a' 



Thus, the theoretically optimal weight to place on 
Tj^j^ increasee as o^g and Sj increase. As mentioned above, 
if as the result of randomization or by luck » 0, then 
both estimates are unbiased and yields the minimum variance 
estimator and becomes simply 

V = ^ S 



^ S + cT^e 



It is interesting to note that this expression represents 
a kind of residual reliability of the test after variation 
related to age and other measurable background variables 
r-n^i^^s been removed. Thus, the weight to be placed on the 
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method which uses the observed pre-^test score Is elmply 
the residual reliability of this score. 

Now suppose we. wish to estimate r^ on the basis of 



the entire treatment group of size nj * Let 



then 



•j2 



E(r^,) « 2a^e 



E{rj2) " + E<ESij) « rj + 

After some calculationr we find that w* which minimizes MSE 
is given by 

» rr2 4. « lis .\ 2 



Note that the relative advantage of Method 1 increases with 
sample size as well as a g and Sj • 
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Suppose now that 5^-0 for all groups. To compute 
w* we need a^^ and a^^ which are not directly available. 
We can estimate w*, however, in two ways, if we have 
some estimate p of reliability, then 

V(T^j) _ V{{a + eajLj + M^^) } + o^^ 



P - 



Prom the regression equations used to produce the residuals, 
we can obtain an estimate of 

V{(o + ea^^ "ij^^ 

A natural estimator of w* is then 

p " 

1 - r2 



Alternatively, one can adopt an empirical approach. 
2 

Let a be the variance of the combined residual with 

w . , 

weight w. From our previous discussion 

2 2 , , ^ , .2 2 

o r, a ( 1 + wi + {1 - w) 0-. 
we s 



If we use these residuals as outcomes, and perform a 

one-way ANOVA with treatments as factors, the meanr square 

2 

error provides an estimate of a . Carrying ouc the ANOVA 

w 
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for ^liferent values of and choosing that valye which 
minimizes the mean square error yields a reasonable 
estimate of For each test, we carried out such a 

procedure, calculating our estimate of w* to the nearest 
tenth. Actually, the minimum can be found ana;LytiQalvly. 
Since the mean square error is a quadratic fmction of w we 
need only calculate its value for any two distinct values 
of w to determine the entire function and Hence the mini- 
mum. It is doubtful, however, that our estimation pro- 
cedure is reliable enough to justify the exact calculation. 
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THEORY UNDERLYING RESISTANT ANALYSIS 

by Sharon Hauck ' 

RESISTANT FITTING TECHNIQUE 

Because Of the use of means, the usual least square)? 
regression estimates will be affected by any extreme 
observations. Therefore i if one does not wish to throw 
aw^y these outliers, but does wish^ less sensitive esti- 
nate, he should look for another method of estimation. 
Tukey^s resistant fitting technique by its use of medians 
serves this purpose. 

The* resistant fitting technique of Tukey may be u6ed 
to fit a model of the form 

/ « a + 3f (Yj^) + e^ (E.l) 

where Y^ and Y^' are the pre and post test scores for 
individual i, e^ is the error term and f is a transforma* 
tion of Y^. Assume there are n individuals. 

We will first find three pairs of representative values 
called summary values. Let us denote these values by (^j, ^ 
where j a 1, 2 and 3. These values will be usdd to find the 
best transformation f and to estimate the slope and the 
intercept. The slope is estimated by . 

i « Y3' - (E.2) 
f(Y3) " f {Y^) 
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and the intercept is estimated by * 

a o T{ E ^.y - gf (^l*.) } (E.3) 

The first step of the procedure involves sorting the 
pair {Y^, Y^") in ascending order on the pre-test score. 
The ordered observations are then divided into thirds of 
approximate size n^/S,^ again according to the pre-test score 
.(Yj. . No ties are broken and, if necessary, the middle 
third will contain the most values. 

The next step is to determine the sunomary values. If 
n is less than thirty, the median pre-test score and the 
median post-*test score in each third serve as the suitonary 
values. (Note, these median scores need not correspond 
to the same individual) . 

\ If n is at least equal to thirty, each thl^d is again 
divided into thirds', forming "Ninths". Within each ninth, 
the median pre and post-test scored are found. For each 
third, its three pairs of ninth median values are then 
averaged to give a pair of ^uunmavy values for that third. 

These three pairs of values are used to help determine 
whether the linear model may be fit on the raw scale 
(i.e., f(Yj^) ^ and, if not, the best r6-expression to 
Induce linearity. This is done by comparirj the upper and 
lower slopes, denoted by Sy and flj^ respectively.* A total 

^* Sy - Y3' - / f(Y3) - fCYj) and Sl = Y2'- Yi' / £(^3) " ^(^1) 
E^t Also, initially f{Y.) = Y . . 
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slope^ S^, Is also used.* 

,If the data is linear, S__ - S_ should be approxi-, 
mately equal to zero. If this difference is not close 
to zero I various re^^expressions or transformations are 
thei) tried. The transformations which ar^ used are of 
the form f(Y^} « kY^^P where k « --1 if p is less than zero 
and k 1 if p is greater than or equal to zero. The 
negation preserves the order of the scores. The case of 
p « 0 corresponds to a natural log re-expression. For the 
PS I and PPV tests^ the following powers were considered: 
-1.5, -1<^ -.5, Or •S, I9 2, 3, 4 and 5. In Tukey's term- 
inology these values form a ladder of powers . 

The configuration of the three summary values or 
the relation between the upper and lower slopes determines 
in what direction one goes on the ladder for trial re- 
expressions. There are four possible configurations as shown 
in Figure 1. If either example A or B is the case, one would 
go down the ladder powers starting at p = .5. The configura- 
tions as shown in C and D call for powers of re-expression 
greater than one.** 



*s^ « ^3> - ^I'/f (ya) - f{^i) 



**In the computer program, the direction was determined b^^ 
comparing to 'X/ 'X; 

If Yj greater than ^ one should go up the ladder of powers 
starting at p = 2. If Yp was less than X, one should go down 
the ladder starting at p = .5. 



ERIC 



335, 




# 336. 



Once the first trial power has been determined, the 
pre-test suitunary values are re-*expressed and the three 
slopes, S^, and are recalculated** As was Indicated 
previously, the transformation which maJtes S„ - the 
closest to zero Is the one to use. However, because the 
various re-expressions cause differences in the magnitudes 
of the slopes, the difference S^, is divided by the 

total slope. Sip, to give a comparable relative difference. 
Therefore, one must look for the smallest relative diff-- 
erence (in absolute value) . 

Another consideration in determining the best trans- 
formation is the sigh of S.-S ♦ If i his difference changes 

u ii 

sign, it indicates one has gone too far on the ladder of 
powers t(over-expressed) ♦ (See Figure 2) ♦ However, if a 
power caused over-expression but still resulted in the 
smallest relative difference it would be used. This is 
because of the .5 or 1 difference between the trial powers. 

After the best re-expiressipn has been found and the % 
slope and intercept have been calculated from the re-expressed 
summary values, the fitting procedure Is completed with an 
examination of the residuals. The quality of the fit can 
be judged by examining a plot of the residuals vs fiY^)* 
The residuals should lie in a band centered around zero. 

>I£ the number of observations in a third or ninth is even, the 
median is the average of two values. In that case, it is necessary 
to re-^express the two values used in calculating the median. The 
re-expressed summary value is then the average of these two values. 
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(Vj, Y3') 



Initial Configuration 




Over-expressed Configuration 



FIGURE 2 

> 
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If a linear trend is apparent from the plot, one may repeat 
the fitting procedure described above, treating [f{)[^), 
residual i] as the data, and omitting the search for a 
re--expression» The new found intercept and slope are then 
added to the original values for a final fit. ^ 

One might also calculate a five number summary of 
the residual values* This svimmary gives the two extreme 
values, the median value and the two hinge values* The 
hinge values are the quartlles and are found from the 
ordered residuals by taking those two values whose ranks 
lie half way between the rank of the median and either ex- 
treme. For example, if there are five observations, the 
hinge values would correspond to those values with ranks 
of two and four. The difference of the hinge values 
approximates the interquartile range of the residual distri- 
bfition. Also, if one believes the residuals to be normally 
distributed, .7 times the difference of the hinge values 
serves as an approximation of the standard deviation.* 

IMPLEMEtyrATION OF THE RESISTANT FITTING TECHNIQUE 

Tables of the results of using this technique for 
the PSI and PPV tests may be found at the end of this section* 
All program Subclasses with more than 5 children were fit. 

* The difference of the hinge values is called the hinge spread. 
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However, In the final comparison only fits based on 20 or 
more children were considered. 

In setting up the computer program to perform the 
resistant fitting procedure, it was necessary to decide which 
powers of re-expression would be considered. The core 
of the ladder, powers from -1 to 3, was chosen because it 
iu the most, commonly used. To allow more freedom, however, 
the ladder was extended from «^1.5 to 5. 

In this context, there is a problem of interpretation 
when the fitted power is greater than 1 — for this reason 
all such fits were not included in the final comparison. 
Strict interpretations of such fits indicate that there 
is no celling effect, i;e., a child who did well on the 
spring test could be expected to do better than 100% in the 
fall. The problem is due to the fact that in some cases 
the model is not complex enough to adequately fit the data. 
We would expect the fitted curve to go to the asymtote 
as in Figure 3A. However, if we fit a curve as shown in 
Figure 3B with a simple polynomial, the resulting fit 
will look like that in Figure 3C. 

In general, the resistant fitting technique allows 
either Yj^ or Y^* or both to be transformed in order to 
induce linearity. We chose only to re-express the pre- 
test scores for sake of interpretation. , 
As was mentioned in Chapter VII, this resistant fitting 
technique could hot be applied to the HRAT subtests. The 
lifficul^y was due to the very large number of zero pre-test 




FIGURE 3B 



J 



— - f(V) 

FIGURE 3C 
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scores and the very small range o£ tQst scores. The 
algorithm that divides the data into thirds does not 
break ties unless there are more than n/3 scores which 
are the same. In that case, only one value Is left in the 
third. For the VJRAT tests this meant the first third only 
contained one obeetrve^tion with Vj^ « 0. And, because so 
many of the pre-test scores were the same, ^2 6qual 
to for mflmy of the programs. This implies was 
infinite and could not be used to determine a re*-expressioh. 

In addition, the small range of test scores led to 
¥3' being equal to Y^* in some cases. This resulted in 
an infinite relative difference which made it impossible 
to determinb a proper re-expression.* 

^Besides, the obvious difficulties discussed above, 
the over-abundance of zero pre-test scores also led to 
l^rge differences between, the upper and lower slopes. The 
following serves an an example: 

WRAT Namln§ Test 
Sponsor: Far West 

White children with no preschool experience 
Sample size JLs 104 

Summary values: (1, 11) (1.33, 2.66) (6.66, 10.66)*** 
Power « 1 * ' , 

Sj^: -25.00 Sy: 1.50 S^: -.06 RDj -450. SO** 



, *If we had been finding fits, we would then have fit a model 
of the form Y^j* a^^ + e^^^ . 

.**RD is. the relative difference. 
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*** In order to ?.t;iry the various re-expresslons, it was 
|/- naoessary to add 1 to, all the pre-test scores. 
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Power « .5 
S^: -53.8 

Power »» 0 
S^: -66.7 

Power = -.5 



S^: -53.87 Syt 5.61 S^i -.21 RD: -282.25 



S, : -66.70 S. 11.45 S_: .24 RD: -193.15 
L U A 



Such large differences between the slopes and very large 
relative differences make any sensible chloice of a re- 
expression otit of the question.* 



*Part of this difficulty is due to the fact that the program 
which breaks the data into thirds does hot order the values 
Therefore, if there is only one value in a third, the post- 
test score may not be the lowest » 
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RESISTANT FITS 

The six sub-classes are coded in following manner: 
the first digit represents ethnicity where "1" is white, 
"a" is Black, and "3" is Mexican -American; and the second 
digit represents prior preschool experience where "0" 
indicates none and "1" indicates some. For example. code 
21 represents Black children with some prior preschool 
experience. 



RESISTANT FITS - PSI TEST 

# Of 

Model , ^ children 

FAR WEST V » 37.93 + 6l.75(-l/Vy) 16? 

TW V « -..32 + 19.13 logy 104 

(20) V' « 26.73 + 80.89(-lA) 20 

(30) y » 28.50 + 106.36(-1/Y) 19 
(11) y - 17.81 + .00 y3 21 

(21) 3 

(31) 0 

ARIZONA Y' « -7.44 + 23.95 log Y 200 

HTT) Y' « 16.01 + .02 y* 86 

(20) Y' « 11.83 + .00 y3 40 

(30) . Y' - 32.67 + 52.00 (-l//y) 8 
(11) Y'. * 3.69 + 20.99 logY 34 

(21) Y' » .10 + 5.06 /7 32 

(31) Y' = 0 

BANK STREET Y' » -7.47 + 22.11 logY 239 

(lb) Y* " 6.47 + .81 Y 19 

(20) Y' « -3.34 + 5.34 Vy 116 

(30) * , 0 
(11) Y' » 16.67 + .00 Y^ 20 

(21) Y' » 8.60 + .61 Y 84 

(31) , 0 
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LogY 155 
f3 9 

(-1//7) ° 51 

/7 66 

3 

Logy 22 

4 

101 

(-l/Y) 39 
LogY 54 

0 
1 

(-1/Y) 7 

yJ 179 

Y* 85 

y2 41 

y5 18 

(rl//7) 14 

y2 - 8 

13 

Y 153 

(-1/Y) 32 

Y- 76 

Yf 21 

y5 6 

(-1/V) 18 

0 

logY 162 

yS 72 
3 ° 

yf -30 

y5 53 

0 

/Y * 114 

(-1//Y) 89 

(20) 0 

(30) 0 
(11) Y' "-11.43+ 29.34 logY 25 

(21) 0 

(31) 0 
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NPV 
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RESISTANT PITS PPV TEST 



Model 
FAR WEST 

— xm 

(20) 
(30) 
(U) 
(21) 
(31) 



Y' 




-16.60 
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41.84 logY 


Y' 
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-19.84 
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43.71 logY 
. .00 Y^ 


Y' 


a 
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Y • 
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9.26 /7 


Y' 
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47.33 


+ 


.00 y 3 



# of ; 
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161 
100 

20 

19 

19 
, 3 
0 • 
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ARIZONA Y' - -1.S6 + 8.07 191 

(10) - Y« « -20.06 + 44.21 logY 85 

(20) Y« - 31.39 + .00 y3 39 

(30) Y' - 28.61 + .03 Y 6 

(11) Y' - 81.10+1303.90 (-1/Y) 31 

(21) Y' - 13.08 + .94 Y 30 

(31) 0 

BANK STREET Y' « -3.97+ 8.09 243 

(10) Y' - 34.62+ .01 Y^ 17 

(2«) Y« -72.46 + 173.94 (-l/»^) 121 

(30) 0 

ill) Y'.« 82.03 + 1358.47 (-1/Y) , 19 

(21) Y" - 77.91 + 197.49 (-1//?) 86 

.(31) 0 

OREGON Y' - 10.26 + 6.12 1^ 136 

(10) Y' - 14.84+ .87.Y . 7 

(20) Y' « 31.52 + ,3$ y 41 

(30) Y' « 12.30 + 5.8(5 * 65 
(U) , 3 

(21) Y' --17.17 + 42.31 logY < 16 

(31) ^ . : 4 

KANSAS Y' - 1.12 + 7.28 V7 98 

(10) YV » 96.99 + 303.36 (-l/»^) 40 

(20) Y' - .64 + 26.50 logY 49 

(30) " 0 

(11) 1 . 

(21) Y' - -21.34 + 10.70 i/7 8 

(31) 0 

HIGH/SCOPE Y' --44.95+ 58.33 logY 168 

(lOJ y' » '21.58 + .68 7 82 

(20) Y' - 10.93 + .94 Y 37 . 

(30) Y' - 39.52 + .00 yS 15 
(11) y - -5.58 + 1.26 y^ 15 

(21) Y« - 25.38 + .00 Y^N \ 7 

(31) y - 43.81 + .00 y4 12 

FLORIDA y » 20.02 + .70 Y 143 

OT) Y' - 97.43 + 290.63 (-l/t/f) 32 

(20) y « -7,82 + 34.05 logY 70 

(30) y - 28.38 + .02 y2 24 
(11) - 4 

(21) y V 29.29 + .00 y^ 13 

(31) ; 0 

EDC V y « -34.21 + 51.85 lo^jY 160 

~ (10) Y* - 64.57 + 554.72 (-1A) 7 

(20) y - 21 .?.4 + .02 y2 70 
o (30) 2. 

pRi/- (11) , y - 63,14 + 533.94 (-1A) 31 

(21) y - 62.07 + 560.87 (-1/Y) 50 

hn ft 
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PITTSBURGH 

— mr" 

(20) 
(30) 
(11) 
(21) 
(31) 

REC 

(10) 
(20) 
(30) 
(11) 
(21) 
(31) 

ENABLERS 

, — rm" 

(20) 
(30) 
(11) 
(21) 
(31) 

CONTROL 

(16) ' 
• (20) 
(30) 
(11) 
(21) 
(31) 



NPV 



(10) 
(20) 
(30) 
(11) 
(21) 
(31) 



Y 



34.83 + .01 
35.36 + .01 y2 



Y' 
Y' 
Y' 



Y' 
Y' 
Y' 
Y' 
Y' 
Y' 
Y' 

Y' 
Y' 
Y' 
Y' 
Y' 
Y' 



Y^ 

yt 

Y' 
Y' 
Y' 

Y' 



35.36 + 



.01 Y' 



24.72 + .63 Y^ 

38.12 + .00 y5 

23.57 + .59 Y 

78.27 +173.17 (-1/7) 



34.27 + 



« — 



-3,30 
32.85 
69.23 
24.74 
75.21 
9.52 
ia.64 

21.37 
32.79 
10.20 
35.18 
39.03 
28.28 



+ 
+ 
+ 
+ 
+ 
+ 
+ 

+ 
+ 
+ 
+ 
+ 
+ 



.43 Y 

8.08 /? 

.01 y2 

170.87 (-1//7) 

.63 Y 
1068.17 (-1//Y) 

.78 Y 

.73 Y 

43,48 logY 

.01 Y^ 

.97 Y. 

.00 Y^ 

54.85 logY 

.00 y5 



22.80 + .63 Y 
11.95 + 5.93 

-.37 + , 7.24 /5r 
56.37 + '220.68 (-1A) 
71.26 + 742.24 (-1/Y) 
35.62 + iOO y1 



•» 27.67 + 



.00 Y" 



111 
85 
0 
0 
26 
0 
0 

71 
12 
18 
31 
0 
4 
6 

202 
72 
63 
32 
23 
6 

106 
30 
32 
14 
11 
15 
4 

629 
166 
223 
96 
25 
106 
13 
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RESISTANT ANALYSIS OF COVARIANCE 

Because" the resistant analysis of covariance tech- 
. nique is i\ew, it is advisable to begin with a brief 

discussion of the motivation behind it.* This is followed 
by a detailed description of the procedure. 

This resistant procedure is analogous to the usual 
or classical one way analysis of covariance. However, 
RANCOVA is designed to be resistant to two kinds of error. 
The first of the^e involves certain observations which may 
be either wrong or wild, i.e., the observation though 
correct, is not a representative member of the population. 
To protect against this, medians are used in finding 
slopes and effects. (See the discussion of the resistant 
fitting technique.) • 

Another problem is that the assumption of equal 
slopes may not be satisfied. Recall in the classical 
Situation, it is assumed that all the treatment means lie 
on parallel lines (as a function of the covariate) , and 
there are tests to judge^ whether this assumption holds. 
There are no such tests for a resistant analysis* In addi- 
tion, the use of a non-interactive computer to perform the 
analysis precludes any use of hume^n judgment concerning thfe 
exclusion of any treatment. TO overcome this second tyjJ^* ' 
of possible error, a weighted mldmean is used instead of a 

♦RANCOVA will be used to denote resistant analysis of covariance. 



351. 



weighted mean vhen combining slopes across tr^aatments.* 

In our usual notation # the model to be fit Is of the 

form 

where Id the covariate and Vj.^' Is the response for 
Individual 1 in treatment ^f^^^ error term, f is 

th 

a predetermined ^e-^expression^ and aj is the j Intercept 
or treatment fit.** Assume there are k treatments and 
Hj Is the number of observations in treatment j. / 

In the classical method, assuming zero means, the 
least squares estimate of & is 
A k nj 

ji"l 1»1 (E.5) / 

j«l 1-1 I 

. - ■ f 

This estimate may be viewed as a weighted mean of the 
classical individual slope estimates Cor each treatment, i.e., 



B - w.j$j ^ 

Ami (E.6) 
Y~ - 

E w. j 



*The choice of a nldmef^n Inetead of a median was made because of 
results from the Princ^iton Study on Robust Estimation! Andrews, 
D.P., et jal. » Robust Sstimates of Location . Princ'etbn/ N. J. : 
Princeton University Press / 1972, ' / 

**He determine f by finding the suisnmary values and going through 
the search procedure as outlined in the first section for each 
program* A compromise re-expression is then used. 
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where w.j • Y..^ and B. » YiaY,.' . , {E.7) 



Under the standard assumption of equal variance, the weighting 
is inversely proportional to the variance of the $j's. 

RANCOVA proceeds analogously by first calculating the 
slopes for each treatment according to Tukey's resistant 
fitting technique, using the specified re-expression. Then, 
the weighted mean of the Pj's is replaced by a weighted 
mldmean . 

Recall the Resistant slope estimate for treatment j 
Is given by (yj» ^ Y^n^Af iY^) - £(Y\^))y Becaiise the 
exact variance of this estimate is not known to this writer, 
an approximation is necessary. Asymptotically, the variance 
of an order statistic, and therefore of and ^3', is pro- 

A 

portlonal to 1/n.* Because of thisi the variance of 3^ will 
be approximately proportional to 



nj{f(^3) - fC^lj^ 
Therefore, weighting Inversely proportional to the variance, a 
weighted sum of slbpes, 



k-r+1 ^ ^ ^ 2 \ ^ 
j„r j 3 1 j 3 1 3^ (E.8) 



(fCYg) - f (Y^))^ 

k-r+1 . . Z 6 

S ^ n.{f(^3) - f{Y,)}/ 
j«r J j 



is used. The sun is over those treatments whose. slopes lie 



♦Wilks, s; S., Mathematical Statistics , New Yorkj Wiley & Sons, 
1962,ip. 273-7T: """" 
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between or are the hinge values.* 

In order td ^he treatment effects, the inter- 

cepts must first be calculated. These values are found 
using Tukey's estimate of 

a. » l/3{ I (? ') . - 0f(Y J^}. (E.9) 
J m-1 J "'J 

Now, rewrite the model as 



^ij' " + + e{f(y)ij - rar. .} + eij**(E.10) 



where p Is the overall medn# :l8 the treatment effeot# and 
f tV) . . is the grand mean of the re-^expressed oovarlate 
values. Comparing this to the original model (E.l), we . 
see that 



and therefore, 



« M + - eTTvTTr (E.il) 

^ : Yj « - {p - enYTT). (B.12) 

Thus, to estimate Yjr one should subtract a' quantity like 
{p - which is constant over all treatments. 

Analogous to the classical situation where En^Yj « 0, we 
win require the median of the Yj*® t6 be zerp, and# 
therefore, will use the median of the a.j*s as an estimate of 
this quantity. The j- treatment effect is then found by ; 
subtracting the median intercept value from a^. \ 

*The subscripts refer to the ordered treatments which have been 
ranked by magnitude of their slopes. 

t{Y),. « f(y, .) ^ 
air i3 i3 
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As with the reaistant fitting technl^que, the analysis 
concludes with an examination of the residuals. In this case 
a plot of the residuals vs. re-expressed covariates for each 
program is useful to judge the goodness of fit. The same 
is true of a five number summary of the residuals for each 
program. 



ERIC 



I 



355. 



IMPLEMENTATION OF RANCOVA . 

A8 was mentioned in Chapter VII ^ only those programs for 
which there were at least 20 children in the sub-^class 
were used in the analysis of the PSI and PPV tests. This 
minimum sample size allowed enough programs for comparison 
while excluding fits based on too few observations. 

RANCOVA could not be used on the WHAT subtests , ETS and 
ITPA tests because of the difficulty in determining re-^expres* 
sions. The reasons for this were discussed in a previous 
section. 

•f 

The choice of a good re-expression f is left up to the 
judgment of the analyst. Because of the interpretation diffi- 
culties related to powers greater than one, we constrained 
the choice of a compromise re-e^pression to lie between -1.5 
and 1 although programs for which p > 1 were allowed, to influ- 
ence the choice, tot example # the following displays the 
number of programs with powers ranging from -1 to 4 for 
White children with no prior preschool experience for the PSl test 



no. of 
programs 




P i 2 power 
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In this case, the choice of power was -.5^ i«e., 
- 5 

f (Y^^) B ""^ij * • eight programs were re-expressed 

accordingly and the analysis continued* ^ 

Left without the classical tests to aid his judgitienti 
the user of resistant methods must rely on plots, displays 
and summary values. To fudge the goodness of fit, we used_ 
plots of the raw and fitted post-test scores toc/ether vb. ^ 
the covariate scores, five number summaries, and hinge- 
spreads of the residuals* By eye, the fitted line should 
••explain*' the data very well, i.e., it should follow the general 
trend of the data. The five number summary should indicate 
symmetric residuals, and the hingespread should be' small 
relative to the range of test scores. 

In a covariance analysis, one expects the treatment 
fits to be parallel or at least very similar. Here again, 
no F test may be performed on the resistant ly determined sflopes. 
A plot of the fits for several programs together is helpful • 
in examining the parallelness of the fits. . Any program whose 
slope is wildly different from the others will be quickly - 
pointed out and one can get an idea of the similarity between ; 
prograitis. 

i Another assumption of the classical ANCOVA is the homo- 
geneity of the variances. To examine this in RANCOVAi one 
may look at a stem and leaf display of the hingespreads of 
the residuals from the initial fit.* Ideally the range of 

*See Tukey (1970) for an explanation of stem and leaf displays. 
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\.tlxQ8e hingespreads should be small. The following is such 

a display fo^ the PSI test^ sub-ciass White children with no 
. prior preschool experience. 



6 I 34 

5 02, 29, 09 
' , 4 25, 02, 25 

• . . 3 85, '50 unit. » .01 

' The values 'in the last line are 3. 85* and 3.50. The .range 
of these hihgespreads is not very. large. 

, . As haft been eWphasi?.ed, plots of the refiiduals from 

. . _ " ^ ■ * . '• 

the final fits were al'so examined. The results of RANCOVA- 
•for the EJSI^ and. ppy, tests way be found in Tables * 
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Appendix F 

INTERPRETATION OF PPV RESULTS * ' 

f * 

From the results of our various analyses^ it seeins 
clear that the performance of the Control children was 
conparable to that of the Head Stuart children / and that no 
particular Head Sta^rt prbgrain was outstanding in raising 
PPV scores. Wte are tempted to .infer . slirply that passive 
vocabulary cind whatever other skills are measured by the ^ 
PPV cannot be influenced very much by preschool curricula. 
If this were indeed the case, we would expect the residual 
analysis to show small residuals fbr both Head Start and . 
Control children. On the contrary i both groups approximately 
double theijr rates of growth.. Thus 'wQ <ur6 left 'to conclude 
either that* there is something misleading about the size 
of the residuals, or that thei growth rate for both groups 
actually did increase over the period studied. 

One possibility i^ that the observed gain9 can be 
attributed to .some sort of test-sdnsitization or practice 
effect. Suppose that for some reason, children find the 
PPV particularly difficult the first time they take it. 
Then if they took it a second time soon after/ they might 



4 
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do a bit better even if their true ability had not changed, 
sinq^ly by virtue of being more used to the test. 

A possible ^pseudo-gain" might also have resulted 
from the fact that the test procedure in the spring was 
slightly different from that in the fall. Children began 
the test with item' 25 Instead of one. If they, answered 
eight in a row correctly/ they were allowed to proceed and 
given credit for the first 24. If not they essentially 
moved back until they were able to correctly a/\8wer eight 
consecutively. Thus it is ^conceivable that a child could 
bo given credit for an item he would have answered in- 
correctly if given the opportunity. By'' looking at the 
nuirbfer answered incorrectly in the fall of those giyen 
credit for in the spring, we obtained a rough upper bound 
on the^pseudo-ga^n attributable to the scoring system. 
Our best guess is that on average one to two points of 
the observed gain may be explained in this way. This still 
leaves us with an average redidual of about four or five - 
points to e3q)lain. 

The fact that the Con tro"!^, children tended to be 
younger than Head Start children ml^Ht als6 be \^ factor. 
Perhaps younger children are g^^ a faster rate^ ^ 

so that we are mderestirtatii^ t^ increments. 
A breakdot^^^^bf residuals by age for the Control children 
.revealed no clear relationship between residual size and age 
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Unfortunately, we see no way of determining whether 
the apparent increase in growth rate is real. The fact that 
there is so little difference among the PV models leads 
us to suspect that the gains are not rolated to program 
characteristics, if programs were effective agents, we 
would expect that, as in the, case of all other tests, at 
least one or two would be particularly effective. Wl,th 
the possible exception oZ REC this does not appear to be * 
the case. W\us, althouc^ we encourage the reader to draw 
his ciwn conclusion from the evidence, we are inclined to 
believe that Head Start programs are ineffective in, ' 
raising PPV scores. ' ^ 
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